Uncategorized

Interfaces in Go

An interface type is a method set. If a type contains methods of the interface, it implements the interface. 

A type can implement multiple interfaces. For instance all type implement the empty interface.

An interface may use a  interface type name  in place of a method specification. This is called embedding interface.

type ReadWriter interface {	
Read(b Buffer) bool
Write(b Buffer) bool
}
type File interface {
ReadWriter // same as adding the methods of ReadWriter
Close()
}

The empty interface

A type of empty interface can hold values of any type.  E.g. For implementing a linkedlist in Golang, we could declare the linkedlist struct as follows

type Node struct {
    Next *Node
    Data interface{}
}

 This allows us to use the same struct to hold data of different types.

n := linkedlist.New(0) // int Data
n.Append(3)
n.Append(9)
for m := n; m != nil; m = m.Next {
    fmt.Println(m.Data)
}
n1 := linkedlist.New("a") //string Data
n1.Append("b")

Useful interfaces in Go stdlib

Error Interface

The error type is an interface that has a method Error.

type error interface {    
Error() string
}

The most commonly used implementation of the error interface is the errorString type in the errors package.

// errorString is a trivial implementation of error.type errorString struct {    
s string
}
func (e *errorString) Error() string {
return e.s
}

Handler Interface

The Handler Interface in the net/http package requires one method ServerHTTP

type Handler interface {	
  ServeHTTP(ResponseWriter, *Request)
}

Within that same package you will find HandlerFunc implements the Handler interface. 

type HandlerFunc func(ResponseWriter, *Request)     

// ServeHTTP calls f(w, r).  
func (f HandlerFunc) ServeHTTP(w ResponseWriter, r *Request) {  
f(w, r)  
}

The HandlerFunc makes it possible for us to pass in any function to make it a Handler. All we would have to do is wrap it in  HandlerFunc. 

http.Handle("/", http.HandlerFunc(indexHandlerHelloWorld))null

We can have also have our own struct that has fields and methods and implements the Handler Interface by defining the ServeHTTP method as a member of the struct. 

Stringer Interface

The fmt package has a stringer interface. It can be implemented by a type that declares the String method. 

type Stringer interface {        
String() string
}

If a type implements the stringer interface, a call to fmt.Println or fmt.Printf of the variable of type will use that method. 

E.g. If you want  to print a struct in a formatted fashion with key names and values, the struct needs to implement the Stringer interface

type Family struct {
    Name string
    Age int
}

func (f *Family) String() string {
    return fmt.Sprintf("Name: %s\t Age: %d", f.Name, f.Age)
}
func main() {
family := []Family{
        {"Alice", 23},
        {"David", 6},
        {"Erik", 2},
        {"Mary", 32},
    }

    for _, i := range family {
        fmt.Println(&i)
    }
}

https://play.golang.org/p/F7rNPyClwG4

The fmt package has other interfaces like Scanner, Formatter and State.

https://golang.org/ref/spec#Interface_types



Golang · Uncategorized

Golang Net HTTP Package

Golang’s net/http package can be used to build a web server in a minutes. It packs in a pretty wide use of Golang concepts like functions, interfaces and types to achieve this.

Here is a basic web server using Go:

package main

import (
	"fmt"
	"net/http"
)

func main() {
	http.HandleFunc("/", handlerHelloWorld)
	http.ListenAndServe(":8082", nil)
}

func handlerHelloWorld(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "Hello world")
}

If we run the above server we can make a GET request and the server will print “Hello World”.

What we need to understand that in the background the package runs a ServeMux to map the url to the handler.

What is ServeMux?

A ServeMux is a HTTP request multiplexer or router that  matches the incoming requests with a set of registered patterns and  calls  the associated handler for that pattern.

http.ListenAndServe has the following signature

func ListenAndServe(addr string, handler Handler) error

If we pass nil as the handler, as we did in or basic server example, the DefaultServeMux will be used.

ServeMux struct contains the following four vital functions that are key to the working of the http package:

func (mux *ServeMux) Handle(pattern string, handler Handler)
func (mux *ServeMux) HandleFunc(pattern string, handler func(ResponseWriter, *Request))
func (mux *ServeMux) Handler(r *Request) (h Handler, pattern string)
func (mux *ServeMux) ServeHTTP(w ResponseWriter, r *Request)

What is a Handler?

Notice that ServeMux has a function named Handler that takes in a reference to a http.Request param and returns a object of type Handler.   Made my head spin a bit when I first saw that.

But looking under the hood, it turns out, http.Handler is simply an interface. Any object can be made a handler as long as it implements the ServeHTTP function with the following signature.

 ServeHTTP(ResponseWriter, *Request)

So essentially the default ServeMux is a type of Handler since it implements ServeHTTP.

HandleFunc and Handle

In our simple server code above, we did not define a Handler that implements ServeHTTP nor did we define a ServeMux. Instead we called HandleFunc and the function that would handle the response.

This is the source code for HandleFunc in the net/http package

func HandleFunc(pattern string, handler func(ResponseWriter, *Request)) {
   	DefaultServeMux.HandleFunc(pattern, handler)
  }  

Internally this calls the DefaultServerMux’s HandleFunc. If you take a look at the implementation of HandleFunc within ServeMux, here is what you’ll find:

func (mux *ServeMux) HandleFunc(pattern string, handler func(ResponseWriter, *Request)) {
 	if handler == nil {
 		panic("http: nil handler")
 	}
  	mux.Handle(pattern, HandlerFunc(handler))
  }

From the net/http source, we find that HandlerFunc type is an adapter to allows the use of an ordinary functions as HTTP handlers.

type HandlerFunc func(ResponseWriter, *Request)
  
   // ServeHTTP calls f(w, r).
  func (f HandlerFunc) ServeHTTP(w ResponseWriter, r *Request) {
  	f(w, r)
  }

The HandlerFunc makes it possible for us to pass in any function to make it a Handler. So in our simple server example above, we could change the HandleFunc call to a call to the Handle function. All we would have to do is wrap it in  HandlerFunc.

http.Handle("/", http.HandlerFunc(indexHandlerHelloWorld))

The Handle function is used when we want to use a custom Handler in our code. 

To demonstrate the use of some of these concepts, here is a simple example of chat server that will receive messages and broadcast them. It uses a Handler that is passed to a ServeMux. 

package main
import (
    "encoding/json"
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
)

func main() {
    mux := http.NewServeMux()
    chatHandler := new(ChatHandler)
    mux.Handle("/ws", chatHandler)
    log.Fatal(http.ListenAndServe(":8080", mux))
}

type MessageDigest struct {
    Text string `json:"message"`
    ToUser string `json:"to"`
}

type ChatHandler struct{}

func (c *ChatHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    if r.Body == nil {
        return
    }
    var msg MessageDigest
    body, err := ioutil.ReadAll(r.Body)
    if err != nil {
        fmt.Fprintf(w, err.Error())
        return
    }
    err = json.Unmarshal(body, &msg)
    if err != nil {
        fmt.Fprintf(w, err.Error())
        return
    }
    fmt.Println("Message for ", msg.ToUser, ": ", msg.Text)
}

 

 

Uncategorized

Channels and Workerpools

Concurrency are part of the golang core. They are similar to light weight threads. We run a routine using the go keyword.

go matchRecordsWithDb(record)

 

Channels

Channels are a way to synchronize and communicate with go routines.

ch := make(chan string)
ch <- "test" //Send data to channel
v := <-ch //receive data from channel and assign to v

The receive here will block till data is available on the channel.

Most programs will use multiple go routines and  buffered channels are vital in synchronizing all the routines

doneCh := make(chan bool, 4)

Here we will be able to run a routine 4 and then it will block till all 4 are received.

Select

The select will block until atleast once case is ready. Select with a default clause is a way to implement non-blocking sends, receives.

WorkerPools

I encountered the classic scenario where I had to make thousands of database calls to match records in a payment file. Finding viable matches in the database per line in the file, was slow and proving to much of a hassle. I wanted to add concurrency to my calls to achieve this faster. However, I was restricted by the database connection. I could only send a set number of queries at a time or it would error out.

I started with the naive approach. I  create a buffered channel of 100 that went out and call the matching routine. The requirement was to match with a key in a table and return results.  Some improvement. It did about 100 queries. Wait for those to finish and start the next batch of 100.

const workers = 100
jobsCh := make(chan int, workers)

for rowIndex := 0; rowIndex < len(fileRecords); rowIndex += workers {
   for j = 0; j < workers; j++ {
      if (rowIndex + j) >=len(fileRecords) {
        break;
      }
      go matchRecordsWithDb(jobsCh,&fileRecords[rowIndex+j])
  } // wait for the 100 workers to return
  for i := 0; i < j; i++ {
      fmt.Printf("%d", <-jobsCh) 
  }
}

There was a major snag in this approach. We had a condition if for some reason the line in the file didn’t have the main key, we had to query on another field. This field was not indexed and took a while to query.  It is a very old legacy system so I can’t change the indexing at this point.

In my entire file I had one such record. The iteration of the 100 workers that had among it the routine to do this one query waited almost a minute and a half on that one query, while the 99 others finished. That is when I started looking at design patterns with channels and came across worker pools.

Worker pools is an approach to concurrency in which a fixed number of m workers have to do n number of  tasks in a work queue. Rather than wait on all the workers (channels) at once, as the workers get idle they can be assigned jobs.

The three main players are :

Collector:  Gathers all the jobs

AvailableWorkers Pool: Buffered channel of channels that is used to process the requests

Dispatcher: Pulls work requests off the Collector and sends them to available channels

All the jobs are add to a collector pool. The dispatcher picks jobs off the collector. If there are availableWorkers it gives them the job else it tries to createOne. If all m workers are busy doing jobs the dispatcher will wait on completion to assign the job.

After reading and experimenting with workerPools, I have written a workerPool package that can be used directly or as a guideline to implement your own.

https://github.com/mariadesouza/workerpool

 

 

 

Uncategorized

Microservices with gRPC

A microservice is an independent runnable services that does one task effectively.  The concept is rather than having one monolithic application, we break it up into independent services that can be easily maintained.

To effectively use microservices there has to be a way for the various independent services to communicate with each other.

There are two ways of communication between microservices:

1. REST, such as JSON or XML over http
2. gRPC – Lightweight RPC protocol brought out by Google

What is gRPC?

To understand gRPC we first take a look at RPC.

RPC(Remote Procedure Call) is a form of inter-process communication (IPC), in that different processes have different address spaces. RPC is a kind of request–response protocol. RPC enables data exchange and invocation of functionality residing in a different address space or process.

gRPC is based around the idea of defining a service, specifying the methods that can be called remotely with their parameters and return types. A client application can call methods on a server application as if it were a local object.

  • gRPC uses the new HTTP 2.0 spec
  • It allows for bidirectional streaming
  • It uses binary rather than text and that helps keep the payload compact and efficient.

This is Google’s announcement for gRPC.

So whats the “g” in gRPC? Google? Per the official FAQ page, gRPC stands for  gRPC Remote Procedure Calls i.e. it is a recursive acronym.

Protocol Buffers

gRPC uses protocol buffers as Interface Definition Language (IDL) for describing both the service interface and the structure of the payload messages.

Protocol buffers are a mechanism for serializing structured data. Define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

https://developers.google.com/protocol-buffers/docs/overview

Specify how you want the information you’re serializing to be structured by defining protocol buffer message types in .proto files. This message is encoded to the protocol buffer binary format.

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;
}

gRPC in Go

go get -u google.golang.org/grpc
go get -u github.com/golang/protobuf/protoc-gen-go

protobuf.Protobuf allows you to define an interface to your service using a developer friendly format.

 

 

 

Uncategorized

String manipulation in Go

Basic Operations

  • Get char array of a String

greeting := "Comment allez-vous"
greeingCharacterArr := []rune(greeting)
  • Get a char at the specific index

fmt.Printf("%c", greeting[5])
fmt.Println(greeting[5])

The result would be  “n“. However, without the character formatter, the result would be the Unicode decimal code 110.

  • Get string length

len(greeting)
  • Substrings
func substring(s string, beginIndex int) string {
 return s[beginIndex:]
}

func substring2(s string, beginIndex int, endIndex int) string {
 return s[beginIndex:endIndex]
}
  • String to Integer
testN,_ := strconv.Atoi("1234") 
testN += 1 
fmt.Println(testN)

strconv package

Above examples : https://play.golang.org/p/kds_Lu9HyTJ

Sorting and Searching Strings in Golang

The sort package in Go has some very useful methods for sorting and searching slices and strings.

The sort.StringSlice attaches the methods of Interface to []string, sorting in increasing order. https://golang.org/pkg/sort/#StringSlice

E.g.

package main
import ( "fmt" "sort")
func main() { 
    countries := sort.StringSlice{"USA", "India", "South africa", "Libya"}
    countries.Sort() 
    n := countries.Search( "USA")
    fmt.Println(""Result: ", n, countries[n])
}

Result: 3 USA

Sorting Characters in a string

To sort characters in a string, we use the fact that strings in Go can be represented as a slice of runes. We could extend the generic sort function in Go to sort the slice of runes. To use the generic sort, we need to implement sort.Interface – Len, Swap, Less.

type sortRunes []rune

func (s sortRunes) Less(i, j int) bool {
    return s[i] < s[j]
}

func (s sortRunes) Len() int{
   return len(s)
}
func (s sortRunes) Swap(i, j int) {
   s[i], s[j] = s[j], s[i]
}

str1 := "sortthisstring"    
s := []rune(str1)
sort.Sort(sortRunes(s))

See the full working example here:

https://play.golang.org/p/f7wFD8NX97d

Note the sort package comments, Sort makes one call to data.Len to determine n, and O(n*log(n)) calls to   data.Less and data.Swap. 

Uncategorized

Embedding in Go

Go supports defining is-a relationships using an embedded type.

Fields and methods in a struct have a has-a relationship

E.g.  Person has-a name and email.

type Person struct {
	Name   string
	Email  string
}

Embedding to define an is-a relationships  is declared with a type but no explicit field name. The unqualified type name acts as the field name.

type Employee struct { 
      Person 
      EmployeeID string 
}

This way the Person struct can be accessed using the type name:

accountant := new(Employee)
accountant.Person.Pay()

We can also invoke any Person methods directly on the Employee object

accountant := new(Employee)
accountant.Pay()

We cannot embed a slice or a map

type Employee struct { 
 []Person
 EmployeeID string 
}

This will give  a syntax error: unexpected [, expecting field name or embedded type

A workaround will be defining a type and then embedding

type SpecialPeople []Person

type Employee struct { 
 SpecialPeople
 EmployeeID string 
}

The spec in Go describes embedding as below:

EmbeddedField = [ "*" ] TypeName .

Embed by-pointer

The advantage of embedding by reference is that  you are embedding all the functionality of a type without needing to know when it is instantiated. The major application of this would be to have thousands of instances sharing a single underlying data structure. This can significantly reduce memory consumption.

type Image struct {
     data [5][5]
}

type Block struct {
   *Image 
   show bool
}

Because, the unqualified type name acts as the field name for an embedded struct, we can’t have an embedded struct and its pointer in the same struct

struct {
	T     // conflicts with embedded field *T and *P.T
	*T    // conflicts with embedded field T and *P.T
	*P.T  // conflicts with embedded field T and *T
}

Promotion of fields or methods

All field or method calls for embedded type objects are resolved at compile-time without the use of a virtual table. A field or method of an embedded field in a struct is called promoted.

type Person struct { 
      Name   string 
      Email  string 
}
func (p *Person) FreeGift() int64{ 
     return 100 
}

type Employee struct {       
     Person       
     EmployeeID string  
}

In this example person.FreeGift()  and employee.FreeGift() is the same so the method is promoted. Promotion  occurs only at the first level.

https://play.golang.org/p/N8rXyIR_CW

However, if we have a method with the same name defined on the parent struct as below, then employee.FreeGift() will invoke the method on the Employee struct

func (e *Employee) FreeGift() int64{ 
      return 200
}

In the above example, calling person.FreeGift() and employee.Person.FreeGift() will invoke the method on the Person struct

https://play.golang.org/p/lYUTijVF7U

One very important design aspect to remember is, if you plan to use an embedded struct and the embedded type has non-exported fields or methods,  those are completely inaccessible to you in a separate package.

Embedding Interfaces

Embedding an interface will add all (exported and non-exported) methods of the embedded interface to the enclosing interface.

type ImageWriter interface { 
      Read(b Buffer) bool 
      Write(b Buffer) bool 
} 
type ImageFile interface { 
     ImageWriter  // same as adding the methods of ImageWriter 
     Close() 
}