Concetto IT

To content | To menu | To search

Development

Entries feed - Comments feed

Thursday 28 January 2016

Null safety in modern programming languages

This is the first part of a blog serie about modern programming languages. This part is about null safety, one of the annoying things in for example the Java programming language. How many times have you seen something like this:


 if(value != null && !value.equals("")) {  
   // do something  
 }  

or ( sometimes called defensive programming...)


 public String foo(String bar) {  
   if (bar == null) {  
     return null;  
   }  
   // do something  
 }  

Today’s IDE’s give you a warning when a variable may be referencing to null but even when you add null checks your code will be less readable, maintainable etc. So, what can we do about it?

Options

Functional programming languages model the absence of a value with an Option type of Maybe type since the start. The Scala language has an Option type and Java 8 now also has an Optional type. The main purpose of the Optional type in Java is to reduce the number of NullPointerExceptions. The combination of Optionals and Lambda expressions is really powerful. When used properly it is hard to produce NullPointerExceptions. The main problem is the readability of the code. For example, we want to check if a persons address has a country:


 public boolean hasCountry(Optional<Person> person) {  
   return person.map(p -> p.address)  
       .map(a -> a.country)  
       .isPresent();  
 }  

Because of backward compatibility the only place in Java 8 were Optionals are used is the Streams API: http://docs.oracle.com/javase/8/docs/api/java/util/class-use/Optional.html Luckily other languages have implemented more advanced mechanisms to prevent NullPointerExceptions.

Safety

In Kotlin, a language developed by Jetbrains, this can be expressed more easily in a one liner. An optional value is marked with a question mark and optionals can be chained:


 class Person {  
   val address: Address?  
   constructor(address: Address?) {  
     this.address = address  
   }  
 }  
 class Address {  
   var country: String?  
   var city: String  
   constructor(country: String?, city: String) {  
     this.country = country  
     this.city = city  
   }  
 }  
 fun hasCountry(person: Person?) : Boolean {  
   return person?.address?.country != null;  
 }  

The Kotlin compiler enforces that you can not pass null values to non-optional variables!

The Swift language is developed by Apple to replace the Objective-C language and claims to be a safe language. The language itself still has notion of null values (nil).


 struct Person {  
   var name : String;  
   var address : Address?  
 }  
 struct Address {  
   var city : String  
   var country: String?  
 }  
 let address = Address(city: "Utrecht", country: nil)  
 let person = Person(name: "test", address: address)  
 print(person.address?.country == nil)  

If we don’t declare the country variable as an optional string the code above will not compile.

The Rust language is designed to be a safe alternative to the C language and has some really cool safety features. For example you can never access a variable that is not initialised. See the first line in the main function, country must have a default value, if we do not declare a default value the code will not compile.


 struct Person {  
  name: String,  
  address: Option<Address>,  
 }  
 struct Address {  
  city: String,  
  country: Option<String>,  
 }  
 fn main() {  
  let address = Address { city: "utrecht".to_string(), country: Default::default()};  
  let person = Person { name: "a person".to_string(), address: Some(address) };  
  println!("Country is set? {}", person.address.unwrap().country.is_some());  
 }  

The compiler of both Rust and Swift is really helpful for the programmer. You have to explicitly declare variables as options in order to use the null reference in Swift. The Rust language does not know about the null reference at all.

More info about:

Kotlin - https://kotlinlang.org/
Swift - https://developer.apple.com/swift/
Rust - https://www.rust-lang.org/

Wednesday 30 September 2015

Microservices and Kerberos authentication

Kerberos authentication is used to secure a variety of big data products like Apache Hadoop and more recently Apache Kafka (>0.9). In case of Hadoop it is used to authenticate each user and service in order to use the Hadoop ecosystem. Also a lot of modern NoSQL databases offer support for Kerberos, for example Apache Cassandra and MongoDB. Kerberos authentication can also be useful in microservice architectures. It can be used to achieve single sign on functionality. A property that is less known is the possibility to delegate tokens to sub services.

Continue reading...

Thursday 2 July 2015

Mapping complex JSON structures with JDK8 Nashorn

How can you map a complex JSON structure to another JSON structure in Java? I think there are a few possible solutions in Java.

The first solution is to use a serialization framework like Jackson, GSON or smart-json. The mapping is a piece of awkward Java code with a lot of if-else conditions. The result is hard to test and hard to maintain. Schematic it looks like this:

JSON -> Java objects -> Mapping -> Java objects -> JSON

An second approach is to use a templating framwork (like Freemarker or Velocity) in combination with a serialization framwork. The logic of the mapping has moved to the template. Schematic it looks like this:

JSON -> Java objects -> Apply template -> JSON

One of the issues with this approach is that the template must enforce that the result is a valid JSON structure. I have tried this approach and it is really hard to produce a valid JSON structure in all use cases.

You could also map your JSON to XML and create the mapping with an XSL transformations. Schematic it looks like this:

JSON -> XML -> XSL transformation -> XML -> JSON

But the ideal schema looks like this:

JSON -> Mapping -> JSON

With JDK 8 and the Nashorn Javascript engine this becomes possible! This implementation provides JSON.parse() and JSON.stringify() by default.

Example Javascript:

function convert(val) {
  var json = JSON.stringify(val);
  var g = JSON.parse(json);
  var d = {
    chunkId: g.chunk.id,
    timestamp: g.chunk.timestamp
  };
  return JSON.stringify(d);
}

Java code:

private ScriptEngineManager engineManager;
private ScriptEngine engine;
public MyConverter() {
  ClassPathResource resource = new ClassPathResource("/converter.js");
  InputStreamReader reader = new InputStreamReader(resource.getInputStream());
  engineManager = new ScriptEngineManager();
  engine = engineManager.getEngineByName("nashorn");
  engine.eval(reader);
}
public String convert(String val){
  return (String) engine.eval("convert(" + source + ")");
}

I think this is -at the moment- the best approach, Java 9 will ship with native JSON support. Perhaps it will become more easier in the future.

Friday 17 April 2015

Uitbreiden van externe library's in Scala

Veel big data frameworks zoals Spark en Kafka zijn geschreven in Scala waardoor ik soms niet de exacte werking weet van een stuk code. In Scala kun je op verschillende manieren programmeren (functioneel, object georiënteerd) dat je niet snel uitgeleerd bent. Zo kwam ik tijdens het gebruik van de Spark Cassandra connector een stuk code tegen dat me nieuwsgierig maakte naar de precieze werking.

Met behulp van de Spark Cassandra connector is het mogelijk om datasets vanuit Cassandra in Spark jobs te gebruiken en weer weg te schrijven in Cassandra. Je kunt bijvoorbeeld een dataset inladen, verschillende aggregaties maken en vervolgens de resultaat set opslaan in een nieuwe tabel.

Allereerst wordt er een Spark configuratie aangemaakt:

1
2
3
4
5
val conf = new SparkConf(true)
.set("spark.cassandra.connection.host", "192.168.123.10")
.set("spark.cassandra.auth.username", "cassandra")
.set("spark.cassandra.auth.password", "cassandra")
val sc = new SparkContext("local[*]", "test", conf)

Een import toegevoegd:

1
import com.datastax.spark.connector._

Hierdoor zijn er extra methodes beschikbaar gekomen op de SparkContext en de RDD’s:

1
2
val rdd = sc.cassandraTable("test", "words")
rdd.toArray.foreach(println)

Hoe is dit mogelijk?

De functies zijn zelfs tijdens het compileren beschikbaar?! In de object georiënteerde taal Java wordt o.a. inheritance (overerving) gebruikt om functionaliteit uit te breiden. Dat is vaak bruikbaar als het om eigen code gaat maar helaas minder bruikbaar als het om externe library’s gaat zoals in bovenstaand voorbeeld. Om deze reden heeft Scala een feature genaamd impliciete type conversies. (implicit type conversions)

Hoe werkt het precies?

Normaal gesproken zou de volgende code niet compileren:

1
val rdd = sc.cassandraTable("test", "words")

De scala compiler is echter zo slim om opzoek te gaan naar functies die een type conversie maken. Doordat we een import statement hebben toegevoegd wordt zo’n type conversies ingeladen:

1
2
3
4
5
package object connector {
implicit def toSparkContextFunctions(sc: SparkContext): SparkContextFunctions =
new SparkContextFunctions(sc)
...
}

En de SparkContextFuncties ziet er als volgt uit:

class SparkContextFunctions(@transient val sc: SparkContext) extends Serializable {
  
  def cassandraTable[T](keyspace: String, table: String)
                       (implicit connector: CassandraConnector = CassandraConnector(sc.getConf),
                        readConf: ReadConf = ReadConf.fromSparkConf(sc.getConf),
                        ct: ClassTag[T], rrf: RowReaderFactory[T],
                        ev: ValidRDDType[T]) =
    new CassandraTableScanRDD[T](sc, connector, keyspace, table, readConf = readConf)

}

Met behulp van impliciete type conversies is het dus mogelijk om externe library’s aan te passen. Het is een krachtig mechanisme om interfaces en methode's toe te kunnen voegen aan bestaande code. Gebruik het echter voorzichtig. De werking is niet direct zichtbaar en je moet naar de import statements kijken om te zien hoe de code precies werkt. En dat laatste doe je niet zo snel.