Golang · Uncategorized

Golang Net HTTP Package

Golang’s net/http package can be used to build a web server in a minutes. It packs in a pretty wide use of Golang concepts like functions, interfaces and types to achieve this.

Here is a basic web server using Go:

package main 
import (
"fmt"
"net/http"
)
func main() {
http.HandleFunc("/", handlerHelloWorld)
http.ListenAndServe(":8082", nil)
}

func handlerHelloWorld(w http.ResponseWriter, r *http.Request){
fmt.Fprintf(w, "Hello world")
}

If we run the above server we can make a GET request and the server will print “Hello World”.

What we need to understand that in the background, the package runs a ServeMux to map the url to the handler.

What is ServeMux?

A ServeMux is a HTTP request multiplexer or router that  matches the incoming requests with a set of registered patterns and  calls  the associated handler for that pattern.

http.ListenAndServe has the following signature

func ListenAndServe(addr string, handler Handler) error

If we pass nil as the handler, as we did in or basic server example, the DefaultServeMux will be used.

ServeMux struct contains the following four vital functions that are key to the working of the http package:

func (mux *ServeMux) Handle(pattern string, handler Handler)
func (mux *ServeMux) HandleFunc(pattern string, handler func(ResponseWriter, *Request))
func (mux *ServeMux) Handler(r *Request) (h Handler, pattern string)
func (mux *ServeMux) ServeHTTP(w ResponseWriter, r *Request)

What is a Handler?

Notice that ServeMux has a function named Handler that takes in a reference to a http.Request param and returns a object of type Handler.   Made my head spin a bit when I first saw that!

But looking under the hood, it turns out, http.Handler is simply an interface. Any object can be made a handler as long as it implements the ServeHTTP function with the following signature.

 ServeHTTP(ResponseWriter, *Request)

So essentially the default ServeMux is a type of Handler since it implements ServeHTTP.

HandleFunc and Handle

In our simple server code above, we did not define a Handler that implements ServeHTTP nor did we define a ServeMux. Instead we called HandleFunc and the function that would handle the response.

This is the source code for HandleFunc in the net/http package

func HandleFunc(pattern string, handler func(ResponseWriter, *Request)) {     
DefaultServeMux.HandleFunc(pattern, handler)
}  

Internally this calls the DefaultServerMux’s HandleFunc. If you take a look at the implementation of HandleFunc within ServeMux, here is what you’ll find:

func (mux *ServeMux) HandleFunc(pattern string, handler func(ResponseWriter, *Request)) {   
if handler == nil {  
panic("http: nil handler")  
}   
mux.Handle(pattern, HandlerFunc(handler))   
}

From the net/http source, we find that HandlerFunc type is an adapter to allows the use of an ordinary functions as HTTP handlers.

type HandlerFunc func(ResponseWriter, *Request)       

// ServeHTTP calls f(w, r).  
 func (f HandlerFunc) ServeHTTP(w ResponseWriter, r *Request) {    f(w, r)   
}

The HandlerFunc makes it possible for us to pass in any function to make it a Handler. So in our simple server example above, we could change the HandleFunc call to a call to the Handle function. All we would have to do is wrap it in  HandlerFunc.

http.Handle("/", http.HandlerFunc(indexHandlerHelloWorld))

The Handle function is used when we want to use a custom Handler in our code. 

To demonstrate the use of some of these concepts, here is a simple example of chat server that will receive messages and broadcast them. It uses a Handler that is passed to a ServeMux. 

package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
)

type MessageDigest struct {
Text string json:"message"
ToUser string json:"to"
}
type ChatHandler struct{}
func (c *ChatHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if r.Body == nil {
return
}
var msg MessageDigest
body, err := ioutil.ReadAll(r.Body)
if err != nil {
http.StatusText(http.StatusInternalServerError),http.StatusInternalServerError)
return
}
err = json.Unmarshal(body, msg)
if err != nil {
http.Error(w, http.StatusText(http.StatusInternalServerError),http.StatusInternalServerError)
return
     }
     w.WriteHeader(http.StatusOK)
fmt.Println("Message for ", msg.ToUser, ": ", msg.Text)
}


func main() {
mux := http.NewServeMux()
chatHandler := new(ChatHandler)
mux.Handle("/ws", chatHandler)
log.Fatal(http.ListenAndServe(":8080", mux))
}
Uncategorized

Channels and Workerpools

Concurrency are part of the Golang core. They are similar to light weight threads. We run a routine using the go keyword.

go matchRecordsWithDb(record)

Channels

Channels are a way to synchronize and communicate with go routines.

ch := make(chan string) 
ch <- "test" //Send data to unbuffered channel
v := <-ch //receive data from channel and assign to v

The receive here will block till data is available on the channel.

Most programs use multiple go routines and  buffered channels.

doneCh := make(chan bool, 4)

Here we will be able to run a routine 4 times without blocking.

select

select is like a switch with cases that allows you to wait on multiple communication operations. It is a way to receive channel data will block until one of its case is ready.  Select with a default clause is a way to implement non-blocking sends, receives.

select {
case ac <- a:
// sent a on ac
case b:= <-bc:
// received b from bc
}

WorkerPools

I encountered the classic scenario where I had to make thousands of database calls to match records from a file to database entries. Finding viable matches in the database per line in the file, was slow and proving to much of a hassle. I wanted to add concurrency to my calls to achieve this faster. However, I was restricted by the database connection. I could only send a set number of queries at a time or it would error out.

I started with the naive approach. I  create a buffered channel of 100 that went out and call the matching routine. The requirement was to match with a key in a table and return results.  Some improvement. It did about 100 queries. Wait for those to finish and start the next batch of 100.

const workers = 100 
jobsCh := make(chan int, workers)
for rowIdx := 0; rowIdx < len(fileRecords); rowIdx += workers {
for j = 0; j < workers; j++ {
if (rowIndex + j) >=len(fileRecords) {
break;
}
go matchRecordsWithDb(jobsCh, &fileRecords[rowIdx+j])
} // wait for the 100 workers to return
for i := 0; i < j; i++ {
fmt.Printf("%d", <-jobsCh)
}
}

There was a major snag in this approach. We had a condition, that if the line in the file didn’t have the main id field, we had to query on another field. This field was not indexed and took a while to query.  It is a very old legacy system so I can’t change the indexing at this point.

In my entire file I had one such record. The iteration of the 100 workers that had among it the routine to do this one query waited almost a minute and a half on that one query, while the 99 others finished. That is when I started looking at design patterns with channels and came across worker pools.

Worker pools is an approach to concurrency in which a fixed number of m workers have to do n number of  tasks in a work queue. Rather than wait on all the workers (channels) at once, as the workers get idle they can be assigned jobs.

The three main players are :

Collector:  Gathers all the jobs

AvailableWorkers Pool: Buffered channel of channels that is used to process the requests

Dispatcher: Pulls work requests off the Collector and sends them to available channels

All the jobs are add to a collector pool. The dispatcher picks jobs off the collector. If there are availableWorkers it gives them the job else it tries to createOne. If all m workers are busy doing jobs the dispatcher will wait on completion to assign the job.

After reading and experimenting with workerPools, I have written a workerPool package that can be used directly or as a guideline to implement your own.

https://github.com/mariadesouza/workerpool

Uncategorized

Microservices with gRPC

A microservice is an independent runnable services that does one task effectively.  The concept is rather than having one monolithic application, we break it up into independent services that can be easily maintained.

To effectively use microservices there has to be a way for the various independent services to communicate with each other.

There are two ways of communication between microservices:

1. REST, such as JSON or XML over http
2. gRPC – Lightweight RPC protocol brought out by Google

What is gRPC?

To understand gRPC we first take a look at RPC.

RPC(Remote Procedure Call) is a form of inter-process communication (IPC), in that different processes have different address spaces. RPC is a kind of request–response protocol. RPC enables data exchange and invocation of functionality residing in a different address space or process.

gRPC is based around the idea of defining a service, specifying the methods that can be called remotely with their parameters and return types. A client application can call methods on a server application as if it were a local object.

  • gRPC uses the new HTTP 2.0 spec
  • It allows for bidirectional streaming
  • It uses binary rather than text and that helps keep the payload compact and efficient.

This is Google’s announcement for gRPC.

So whats the “g” in gRPC? Google? Per the official FAQ page, gRPC stands for  gRPC Remote Procedure Calls i.e. it is a recursive acronym.

Protocol Buffers

gRPC uses protocol buffers as Interface Definition Language (IDL) for describing both the service interface and the structure of the payload messages.

Protocol buffers are a mechanism for serializing structured data. Define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

https://developers.google.com/protocol-buffers/docs/overview

Specify how you want the information you’re serializing to be structured by defining protocol buffer message types in .proto files. This message is encoded to the protocol buffer binary format.

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;
}

gRPC in Go

go get -u google.golang.org/grpc
go get -u github.com/golang/protobuf/protoc-gen-go

protobuf.Protobuf allows you to define an interface to your service using a developer friendly format.

Uncategorized

String manipulation in Go

Basic Operations

  • Get char array of a String
greeting := "Comment allez-vous" 
greeingCharacterArr := []rune(greeting)
  • Get a char at the specific index
fmt.Printf("%c", greeting[5])
fmt.Println(greeting[5])

The result would be  “n“. However, without the character formatter, the result would be the Unicode decimal code 110.

  • Get string length
len(greeting)
  • Substrings
func substring(s string, beginIndex int) string{
return s[beginIndex:]
}

func substring2(s string, beginIdx int, endIdx int) string {
return s[beginIdx:endIdx]
}
  • String to Integer
testN,_ := strconv.Atoi("1234")  
testN += 1
fmt.Println(testN)

strconv package

Above examples : https://play.golang.org/p/kds_Lu9HyTJ

  • Comparing Strings

To compare strings, use the comparison operators ==, < and >

The strings package has a Compare function that comes with a warning that no one should use strings.Compare

https://golang.org/src/strings/compare.go

Sorting and Searching Strings in Golang

The sort package in Go has some very useful methods for sorting and searching slices and strings.

The sort.StringSlice attaches the methods of Interface to []string, sorting in increasing order. https://golang.org/pkg/sort/#StringSlice

E.g.

package main 
import (
"fmt"
"sort"
)

func main() {
countries := sort.StringSlice{"USA", "India", "South Africa"}
countries.Sort()
n := countries.Search( "USA")
fmt.Println(""Result: ", n, countries[n])

}


Result: 3 USA

Sorting Characters in a string

To sort characters in a string, we use the fact that strings in Go can be represented as a slice of runes. We could extend the generic sort function in Go to sort the slice of runes. To use the generic sort, we need to implement sort.Interface – Len, Swap, Less.

type sortRunes []rune 

func (s sortRunes) Less(i, j int) bool {
return s[i] < s[j]
}

func (s sortRunes) Len() int{
return len(s)
}

func (s sortRunes) Swap(i, j int) {
s[i], s[j] = s[j], s[i]
}

str1 := "sortthisstring"
s := []rune(str1) sort.Sort(sortRunes(s))

See the full working example here:

https://play.golang.org/p/f7wFD8NX97d

Note the sort package comments, Sort makes one call to data.Len to determine n, and O(n*log(n)) calls to   data.Less and data.Swap. 

Uncategorized

Embedding in Go

Go supports defining is-a relationships using an embedded type.

Fields and methods in a struct have a has-a relationship

E.g.  Person has-a name and email.

type Person struct { 
Name string
Email string
}

Embedding to define an is-a relationships  is declared with a type but no explicit field name. The unqualified type name acts as the field name.

type Employee struct {        
Person
EmployeeID string
}

This way the Person struct can be accessed using the type name:

accountant := new(Employee) 
accountant.Person.Pay()

We can also invoke any Person methods directly on the Employee object

accountant := new(Employee) 
accountant.Pay()

We cannot embed a slice or a map

type Employee struct {   
[]Person
EmployeeID string
}

This will give  a syntax error: unexpected [, expecting field name or embedded type.

A workaround will be defining a type and then embedding :

type SpecialPeople []Person 
type Employee struct {
SpecialPeople
EmployeeID string
}

The spec in Go describes embedding as below:

EmbeddedField = [ "*" ] TypeName .

Embed by-pointer

The advantage of embedding by reference is that  you are embedding all the functionality of a type without needing to know when it is instantiated. The major application of this would be to have thousands of instances sharing a single underlying data structure. This can significantly reduce memory consumption.

type Image struct {      
data [5][5]
}
type Block struct {
*Image
show bool
}

Because, the unqualified type name acts as the field name for an embedded struct, we can’t have an embedded struct and its pointer in the same struct.

type Image struct { 
T // conflicts with embedded field *T and *P.T
*T // conflicts with embedded field T and *P.T
*P.T // conflicts with embedded field T and *T
}

Promotion of fields or methods

All field or method calls for embedded type objects are resolved at compile-time without the use of a virtual table. A field or method of an embedded field in a struct is called promoted.

type Person struct {        
Name   string
Email  string
}
func (p *Person) FreeGift() int64{
return 100
}
type Employee struct {      
Person      
EmployeeID string 
}

In this example person.FreeGift()  and employee.FreeGift() is the same so the method is promoted. Promotion  occurs only at the first level.

https://play.golang.org/p/N8rXyIR_CW

However, if we have a method with the same name defined on the parent struct as below, then employee.FreeGift() will invoke the method on the Employee struct.

func (e *Employee) FreeGift() int64{        
return 200
}

In the above example, calling person.FreeGift() and employee.Person.FreeGift() will invoke the method on the Person struct

https://play.golang.org/p/lYUTijVF7U

One very important design aspect to remember is, if you plan to use an embedded struct and the embedded type has non-exported fields or methods,  those are completely inaccessible to you in a separate package.

Embedding Interfaces

Embedding an interface will add all (exported and non-exported) methods of the embedded interface to the enclosing interface.

type ImageWriter interface {        
Read(b Buffer) bool
Write(b Buffer) bool
}
type ImageFile interface {
ImageWriter  // same as adding methods of ImageWriter
Close()
}

Check out this example to add logging to your struct by embedding the log.Logger object.

https://play.golang.org/p/orTWK_TaM_W

Uncategorized

Go Data Types

Basic data types

bool 
string
int int8 int16 int32 int64
uint uint8 uint16 uint32 uint64 uintptr
byte // same as for uint8
rune // represents a Unicode code point
float32 float64
complex64 complex128

int

int is a signed integer type that is at least 32 bits in size. It is a distinct type, however, and not an alias for, say, int32

An int is either 32 or 64 bits depending on the platform. So it maybe better to be specific if developing and host machines differ.

Rune

Strings are made up of “runes” rather than chars. Rather than having char and wchar for characters and unicode characters, a rune is a “code point” as defined in the Unicode standard. Internally it is  a 32-bit representation.

nihao := "你好" 
for index, runeValue := range nihao {
fmt.Printf("Char %#U Byte position %d\n", runeValue, index)
}
  • String to array of runes
runes := []rune(s)
  • rune array to string
str := string(runes)

Simple Composite Types

Slice

A slice consists of a pointer to an array segment, length of the segment and its capacity i.e. the maximum length of the segment.

  • Create
b = make([]T, len(a))

Note that since the zero value of a slice is nil, you don’t need to explicitly make a slice, you can just declare it like below.

  • Add element
var s []int      // s := make([]string, 0) 
s = append(s, 1) //add one element
s = append(s, 2,3,4) //add multiple
s = append(s, s1…) // append another slice
  • Delete element
s = s[:len(s)-1] //remove last element

Be aware that if a slice contains pointers to elements and a delete is done, there maybe a potential memory leak. Values referenced by the removed slice elements may not be garbage collected as the references are not totally removed.

  • Remove all elements of a slice

To remove all elements of a slice, set it to nil. This will release it to the garbage collector as long as there no other references to it.

s = nil
  • Iterate over a slice
for index, value := range s { 
fmt.Println(index, value)
}
  • Copy

Use copy to deep copy all slice elements from one slice to another. Simply assigning one slice will only change the reference.

a := []int{1, 2}
b := []int{3, 4}

a = b // changes a to refer to the location of b

copy(a,b) //actually copy values from b to a

https://play.golang.org/p/X_WC1eN7IHCh

The number of elements copied is the minimum of len(src) and len(dst). For example in the below code the destination slice will have no elements since len(dst) is 0

s := []int{4, 2,3, 1}
var dst []int
copy(dst, s)
fmt.Println(dst)
fmt.Println(s)

Output:
[]
[4 2 3 1]

We need to allocate a length when creating the destination slice

dst := make([]int, len(s)
copy(dst, s)

https://play.golang.org/p/Sp4NoX4YaaA

Two dimensional slices

Slices in Go are one-dimensional. We have to compose it to form two-dimensional arrays.

matrix := make([][]int, row) 
for m := range matrix {
matrix[m] = make([]int, col)
}

When you pass a slice to a function, since it is a pointer to an array you just get a copy of the slice structure. It will still point to the same underlying array segment. So any modifications made to the slice within the function will be seen outside.

https://play.golang.org/p/OLkoxGf8uaZ

However, if you append an element, remember that a new slice is created and elements are copied over so you will lose the elements if this happens within a function so you must return a slice. E.g. append from the stdlib returns a new slice.

  • Creating and initializing a 2d slice
graph := [][]int{  
{0, 4, 0, 0, 0},
{4, 0, 8, 0, 0},
{0, 8, 0, 7, 0},
{0, 0, 7, 0, 9},
{0, 0, 0, 9, 0},
}

Map

The built-in map type implements a hash table.


Maps, like channels, but unlike slices, are just pointers to runtime types.

Map is a pointer to a runtime.hmap structure. So when we declare a map using var, its value will be nil until initialized using make. Attempts to access a nil map will result in a panic.

Maps are not concurrent safe. So proceed with caution when reading/writing to them in more than one go routine.

  • Create
x := make(map[int]string)

You could initialize a map with values. In that case, you don’t have to use make

colorCodes := map[string]string{  
"BRONZE":   "E0001B",
"GOLD":    "A89968",
"SILVER":   "888B8D",
}
fmt.Println(colorCodes["BRONZE"])
  • Add element to a map
x[1] = 10
  • Delete element
delete(x, 1)
  • Find value
val, ok := x[key]
  • Iterating
for key, val:= range x {     
println("Key:", key, "Value:", val ,"Value:", m[k])
}
  • Size
len(m)
  • Copy

To copy a map to another, we need to traverse the map and do a deep copy. Just assigning a map to another will not copy contents over.

https://play.golang.org/p/JiuMeE3rClf

All operations on a map are in constant time. Most map operations like lookup, insertion etc. are rewritten at compile time to runtime calls. 

You don’t need to use references with a map. Map types are reference types like slices.  If you pass a map to a function and add key-value pairs or update values , it will be reflected outside a function. When using maps in concurrent routines, we need to add locks to ensure data integrity.

https://play.golang.org/p/j7dD1K_rfRY

Struct Type

struct is a collection of fields.

A struct can have methods defined on it. There are two kinds of methods that can be defined on a  struct, a value receiver and a pointer receiver.

type T struct { 
a int
}

func (tv T) Mv(a int) int {
return 0
} // value receiver

func (tp *T) Mp(f float32) float32 {
return 1
} // pointer receiver

It all comes down which one is better for the problem at hand.

Related blog posts for Structs:

Constants

There are boolean constantsrune constantsinteger constantsfloating-point constantscomplex constants, and string constants. Rune, integer, floating-point, and complex constants are collectively called numeric constants.

A constant expression may contain only constant operands and are evaluated at compile time. As such, there is no such thing as array constant in Go. Slices and arrays are always evaluated during runtime. They can only be defined as variables.

Auto increment constants can be defined using iota.

const (
Black = iota
White
Red
Yellow
Green
)

fmt.Println(Black, White, Red, Yellow, Green)

Output:
0 1 2 3 4

Equality of data types

Simple data types and individual elements can be compared using the equality operator ==

Arrays and slices can only be compared to nil.

For other complex datatypes we can use the DeepEqual function in the reflect package.

package main
import (
"fmt"
"reflect"
)

func main(){
a := []int{3,4,5}
b := []int{3,4}
b = append(b, 5)
if reflect.DeepEqual(a,b){
fmt.Println("slices a and b are equal")
}
}

Struct values can also be compared using DeepEqual. The exported as well as unexported values are compared for equality.

Channels can be compared like numbers and strings using the == operator.

Related:

Interface Types