Golang

Auto-generate code using Go templates

The Golang template package is very useful in generating custom code especially if the code is very similar but needs multiple tweaks to work for different platforms or products.

I have used Golang templates for code generation in the following scenarios:

  1. Generating multiple Chromium and Firefox extensions for varied products.
  2. Interfacing with different API’s to get data into our main systems

I’m also looking into using it to auto generate custom mobile apps.

Here, I will explain in detail how I build new chrome extensions in seconds using Golang templates.

There is a detailed description on how to build a chrome extension. Once you have a skeletal chrome extension, it is easy to duplicate and create multiple custom extensions if needed or even allow to build Firefox extension using the same set of extension files.

All you need is a set of template files and your config json values.

E.g. The chrome extension manifest.json.template file will look like this with template fields:

{
 "name": "{{.Name}}",
 "version": "{{.Version}}",
 "manifest_version": 2,
 "default_locale": "en",
 "description": "{{.Description}}",  
 "background": {  "page": "background.html"    },  
 "browser_action": {      "default_title": "{{.Tooltip}}",   
 "default_icon": "icon.png"          },
"icons": { "16": "icon16.png",
            "48": "icon48.png",        
            "128": "icon128.png"     
},  
"homepage_url": "{{.Protocol}}://{{.Domain}}",  
"permissions": [    "tabs",      
"{{.Protocol}}://{{.Domain}}/"  
]
}

Similarly we write templates file for all the extension files like popup.js etc.

To build a basic chrome extension, I define the following in a global.json file

{
 "production": "true",
 "author": "Maria De Souza",
 "author_home": "https://mariadesouza.com/",
 "protocol": "https",
 "version" : "0.0.0.1",
 "domain" : "www.mysite.com",
 "name" : "testExtension",
 "description" : "This is a test extension",
 "tooltip":"Click here to view settings",
 "title" : "Test extension",
}

These settings can be overridden by a product specific json file:

{
"title" : "My cool new extension",
"version" : "0.0.0.2",
}

The product.json can be a subset of the original config.

Now we can get to fun part, building a script to generate the extensions. We first define a struct to unmarshal our config json and use it in our build script.

type Config struct { 
Production  string `json:"production,omitempty"` 
Author      string `json:"author,omitempty"` 
AuthorHome  string `json:"author-home,omitempty"` 
Version     string `json:"version,omitempty"` 
Domain      string `json:"domain,omitempty"` 
Protocol    string `json:"protocol,omitempty"` 
Name        string `json:"name,omitempty"` 
Description string `json:"description,omitempty"` 
Tooltip     string `json:"tooltip,omitempty"` 
Title       string `json:"title,omitempty"` 
Browser     string `json:"browser,omitempty"` 
ProductDir  string `json:"product_dir,omitempty"` 
HomePage    string `json:"home_page,omitempty"` 
UpdateURL   string `json:"update-url,omitempty"`
}

Start by unmarshalling the global file in a struct value as below. I have left out error handling to reduce noise. We then unmarshal the custom product values.

var globalConfig Config
configFile, _ := ioutil.ReadFile("global.json") 
json.Unmarshal(configFile, &globalConfig)
var productConfig Config
productconfigFile,_ := ioutil.ReadFile("product.json") 
json.Unmarshal(productconfigFile, &productConfig)

Using reflect, I override the custom product values:

func mergeWithGlobal(destConfig, srcConfig *Config){
 st := reflect.TypeOf(*destConfig) 
 for i := 0; i < st.NumField(); i++ { 
  tag := strings.Split(field.Tag.Get("json"), ",") 
  v2 := reflect.ValueOf(destConfig).Elem().FieldByName(st.Field(i).Name) 
  if tag[0] != "" && v2.String() != "" { 
   v := reflect.ValueOf(srcConfig).Elem().FieldByName(st.Field(i).Name) 
   v.SetString(v2.String()) 
  } 
 } 
}

Using the Config struct, I then populate the template files. To do this I read all files with extension .template in the source diectory, execute the template using the populated Config struct and save the result in the destination directory.

func populateTemplateFiles(source, destination string, globalConfig *Config) error { 
 templatefiles, _ := ioutil.ReadDir(source) 
 re := regexp.MustCompile(`(.*)\.template$`) 
 os.MkdirAll(destination, 0755) 
 for _, file := range templatefiles { 
  if re.MatchString(file.Name() ) == true { 
   buf, _ := ioutil.ReadFile(filepath.Join(source,file.Name())) 
   tmpl, _ := template.New("extensions").Parse(string(buf)) 
   targetfilename := strings.Split(file.Name(), ".template") 
   targetfile := filepath.Join(destination, targetfilename[0] ) 
   f, _ := os.OpenFile(targetfile, os.O_WRONLY|os.O_CREATE, 0755) 
   w := bufio.NewWriter(f) 
   tmpl.Execute(w, globalConfig) 
   w.Flush() 
  } 
 } 
return nil
}

I also save customized images in the product directory. This way in the build script we can copy custom images in the destination directory. We can then upload a zipped version to the chrome webstore.

 

Golang

Launch a Golang web server using Docker

If you want to create a web server using Go the simplest way to deploy is using Docker.  Golang code is compiled to a binary and does not need a special environment to run.

Here is the simplest web server code in Go to get started. Save this as webserver.go

package main
import ( 
      "fmt" 
      "log" 
      "net/http" 
      "runtime"
     )
func main() { 
     http.HandleFunc("/", indexHandlerHelloWorld) 
     log.Fatal(http.ListenAndServe(":8080", nil))
}

func indexHandlerHelloWorld(w http.ResponseWriter, r *http.Request) { 
     fmt.Fprintf(w, "Hello world, I'm running search on %s with an %s CPU ", runtime.GOOS, runtime.GOARCH)
}

We can use the simplest docker image scratch and add a directive to copy the binary to the server. Save as Dockerfile.

FROM scratch
MAINTAINER Maria De Souza <maria.g.desouza@gmail.com>

ADD go-webserver go-webserver
ENTRYPOINT ["/go-webserver"]
EXPOSE 8080

We setup a start shell script to build and kick up the docker

#!/bin/bash

GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -a -o go-webserver webserver.go || exit 1
if [ "$(docker ps -q -f name=go-web-server)" ]; then
 docker stop $(docker ps -a -q --filter name=go-web-server --format="{{.ID}}")
fi
docker build -t go-web-server . || exit 1
docker run -p 8000:8080 go-web-server || exit 1

You can use the -h option to add the hostname to the webserver

Voila! Now your webserver is running. Navigate to http://localhost:8000/ to test.

Known Issue

If you make any SSL requests from your webserver, you will see the following error when running the webserver using Docker:

x509: failed to load system roots and no roots provided

This is because the /etc/ssl/certs/ca-certificates.crt, is missing from the scratch image that is read by the golang tls package.

To avoid this copy the cert to your docker image from your local system. I normally add this to my bash script that will copy it based on OS.

if [ ! -e ca-certificates.crt ]; then
 if [[ $(uname) = "Darwin" ]]; then
 cp /usr/local/etc/openssl/cert.pem ca-certificates.crt || exit 1
 else 
 cp /etc/ssl/certs/ca-certificates.crt ca-certificates.crt || exit 1
 fi
fi

This is what your Dockerfile should then look like:

FROM scratch 
MAINTAINER Maria De Souza <maria.g.desouza@gmail.com> 

COPY ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
ADD go-webserver go-webserver 
ENTRYPOINT ["/go-webserver"] 
EXPOSE 8080

 

 

 

Tech News · Technology Security

Meltdown and Spectre Security Flaw

Although most of the news sites initially reported the Meltdown and Spectre security flaws as present in an Intel processor, it is now known to affect other processors like AMD and ARM as well. That means all  devices using these processors like PCs, MacBooks, servers, Android and iOS devices are affected.

What are the Spectre and Meltdown security flaws?

These are security holes introduced by two different optimization techniques used by the processors namely: Speculative Execution and Out-of-order execution.

The technique used to exploit Speculative Execution is called Spectre and has two variations. One that takes advantage of the bounds checks bypass and the other that exploits the capability to do branch target injection by altering the branch target buffer to execute the rogue process.

The technique use to exploit the out-of-order execution performance feature is called Meltdown. This enables a rogue process to read memory of another process or virtual machine in the cloud without permission or privileges.

How to protect yourselves?

Most of the tech industry giants have responded quick. Google in particular developed a mitigation technique to protect against Spectre and shared it with other partners.

Android

The Android 2018-01-05 Security Patch Level(SPL) includes mitigations reducing access to high precision timers that limit attacks on all known variants on ARM processors. These changes were released to Android partners in December 2017

Chromebooks

OS versions prior to 63 are not patched. Chrome OS systems started receiving version 63 on 12/15/2017.

Go to the Google FAQ for steps to take on Google cloud and other Google products.

Linux

After being vocal in his criticism of Intel’s responses to the problems, Linus Torvalds released the first new Linux kernel of 2018 on Jan. 28, after the longest development cycle for a new Linux kernel in seven years. Linux 4.15 was released with improved Meltdown, Spectre Patches. Read the release announcement from Linus Torvalds here.

Microsoft Windows 10

After Intel’s Buggy fix to Spectre, Microsoft has issued a patch to fix this.

Microsoft  has announced more security updates for Windows 10 devices in its March Patch Tuesday. These can be found here. They have also lifted the AV compatibility check put in earlier on Anti Virus software that made calls to the kernel memory.

Apple Devices: MacBook, iPhone, iPad, Apple TV

As of Jan 4th, Apple confirmed that it has addressed the recent “Meltdown” as well as Spectre  vulnerability in previously released iOS 11.2.2, macOS High Sierra 10.13.2, and tvOS 11.2

Browsers

Firefox 57.0.4 released on Jan 4, 2018 includes the two mitigations

Microsoft Edge: Microsoft has released an update to Windows Client to fix the vulnerability on Edge(KB4056890). Check details.

Chrome 64, due to be released on January 23, will contain mitigations to protect against exploitation.

Safari Apple has released new security updates aimed at protecting Safari and WebKit from the Spectre attack. Check details here.

Amazon Cloud

Amazon Web Services(AWS) team put out a security bulletin on Jan 03, 2018 with instructions for customers to follow on protecting their servers against the vulnerability.

These are only the first set of software mitigations. With increasing pressure, Intel admitted that these updates do not totally eliminate the risks. They are now implementing hardware mitigations directly into their chips.

Uncategorized

String manipulation in Go

Basic Operations

  • Get char array of a String

greeting := "Comment allez-vous"
greeingCharacterArr := []rune(greeting)
  • Get a char at the specific index

fmt.Printf("%c", greeting[5])
fmt.Println(greeting[5])

The result would be  “n“. However, without the character formatter, the result would be the Unicode decimal code 110.

  • Get string length

len(greeting)
  • Substrings
func substring(s string, beginIndex int) string {
 return s[beginIndex:]
}

func substring2(s string, beginIndex int, endIndex int) string {
 return s[beginIndex:endIndex]
}
  • String to Integer
testN,_ := strconv.Atoi("1234") 
testN += 1 
fmt.Println(testN)

strconv package

Above examples : https://play.golang.org/p/kds_Lu9HyTJ

Sorting and Searching Strings in Golang

The sort package in Go has some very useful methods for sorting and searching slices and strings.

The sort.StringSlice attaches the methods of Interface to []string, sorting in increasing order. https://golang.org/pkg/sort/#StringSlice

E.g.

package main
import ( "fmt" "sort")
func main() { 
    countries := sort.StringSlice{"USA", "India", "South africa", "Libya"}
    countries.Sort() 
    n := countries.Search( "USA")
    fmt.Println(""Result: ", n, countries[n])
}

Result: 3 USA

Sorting Characters in a string

To sort characters in a string, we use the fact that strings in Go can be represented as a slice of runes. We could extend the generic sort function in Go to sort the slice of runes. To use the generic sort, we need to implement sort.Interface – Len, Swap, Less.

type sortRunes []rune

func (s sortRunes) Less(i, j int) bool {
    return s[i] < s[j]
}

func (s sortRunes) Len() int{
   return len(s)
}
func (s sortRunes) Swap(i, j int) {
   s[i], s[j] = s[j], s[i]
}

str1 := "sortthisstring"    
s := []rune(str1)
sort.Sort(sortRunes(s))

See the full working example here:

https://play.golang.org/p/f7wFD8NX97d

Note the sort package comments, Sort makes one call to data.Len to determine n, and O(n*log(n)) calls to   data.Less and data.Swap. 

Uncategorized

Embedding in Go

Go supports defining is-a relationships using an embedded type.

Fields and methods in a struct have a has-a relationship

E.g.  Person has-a name and email.

type Person struct { 
Name string
Email string
}

Embedding to define an is-a relationships  is declared with a type but no explicit field name. The unqualified type name acts as the field name.

type Employee struct {        
Person
EmployeeID string
}

This way the Person struct can be accessed using the type name:

accountant := new(Employee) 
accountant.Person.Pay()

We can also invoke any Person methods directly on the Employee object

accountant := new(Employee) 
accountant.Pay()

We cannot embed a slice or a map

type Employee struct {   
[]Person
EmployeeID string
}

This will give  a syntax error: unexpected [, expecting field name or embedded type.

A workaround will be defining a type and then embedding :

type SpecialPeople []Person 
type Employee struct {
SpecialPeople
EmployeeID string
}

The spec in Go describes embedding as below:

EmbeddedField = [ "*" ] TypeName .

Embed by-pointer

The advantage of embedding by reference is that  you are embedding all the functionality of a type without needing to know when it is instantiated. The major application of this would be to have thousands of instances sharing a single underlying data structure. This can significantly reduce memory consumption.

type Image struct {      
data [5][5]
}
type Block struct {
*Image
show bool
}

Because, the unqualified type name acts as the field name for an embedded struct, we can’t have an embedded struct and its pointer in the same struct.

type Image struct { 
T // conflicts with embedded field *T and *P.T
*T // conflicts with embedded field T and *P.T
*P.T // conflicts with embedded field T and *T
}

Promotion of fields or methods

All field or method calls for embedded type objects are resolved at compile-time without the use of a virtual table. A field or method of an embedded field in a struct is called promoted.

type Person struct {        
Name   string
Email  string
}
func (p *Person) FreeGift() int64{
return 100
}
type Employee struct {      
Person      
EmployeeID string 
}

In this example person.FreeGift()  and employee.FreeGift() is the same so the method is promoted. Promotion  occurs only at the first level.

https://play.golang.org/p/N8rXyIR_CW

However, if we have a method with the same name defined on the parent struct as below, then employee.FreeGift() will invoke the method on the Employee struct.

func (e *Employee) FreeGift() int64{        
return 200
}

In the above example, calling person.FreeGift() and employee.Person.FreeGift() will invoke the method on the Person struct

https://play.golang.org/p/lYUTijVF7U

One very important design aspect to remember is, if you plan to use an embedded struct and the embedded type has non-exported fields or methods,  those are completely inaccessible to you in a separate package.

Embedding Interfaces

Embedding an interface will add all (exported and non-exported) methods of the embedded interface to the enclosing interface.

type ImageWriter interface {        
Read(b Buffer) bool
Write(b Buffer) bool
}
type ImageFile interface {
ImageWriter  // same as adding methods of ImageWriter
Close()
}

Check out this example to add logging to your struct by embedding the log.Logger object.

https://play.golang.org/p/orTWK_TaM_W

Golang

Packages on the Go

Package management with Go is a very talked about issue. Unfortunately, go get does not support functionality to fetch specific tags or versions. It gets the package from the HEAD in git.

Recently we had a situation where a developer on our team used a package and then it got obsoleted. The package developer tagged the release before making breaking changes and the sources were still available but we had to do a git checkout and run it.

That is when I looked into package management. Go 1.5 introduced the “vendor” directory as an experiment and made it official  in Go 1.6

If you use third party packages in your product, copy it to the vendor directory and go searches for dependencies there.

Ex.

package main 

import (
   "fmt"
   "io"

   "https://github.com/mariadesouza/sftphelper" 
)

 

main.go
vendor
 |--github.com
 |     |--mariadesouza
 |     |    |-- sftphelper
 |     |    |     |-- LICENSE
 |     |    |     |-- README.md 
 |     |    |     |--sftphelper.go
Golang

Pass by value or reference

The official Go site FAQ states,  “As in all languages in the C family, everything in Go is passed by value”. This is because the function gets a copy of everything that is passed in.

Is there such thing as pass by reference in Go?

There are different views  as to what is exactly pass by reference to Go.  Some strongly maintain there is no such thing as pass by reference. In C++ terms, the actual meaning of pass by reference is you pass a reference or a pointer to the actual data structure rather the data itself. The function then can modify the value of the argument using that reference.

In Go when I pass a pointer to a struct for example, whether its a copy to the pointer or not, I am not passing the struct itself but a pointer or a reference to it.  I can modify the actual struct using the pointer. In my view, that fits the definition of pass by reference.

When to pass a pointer?

We don’t need to pass pointers to map and slices  as they are already descriptors that contain pointers to the actual map or slice data.

Compelling arguments to use pointer receiver and pass by reference:

  • You want to modify the receiver. With value receivers you can’t modify the struct itself
  • Its is a big struct. It will cost to deep copy the struct.

When you pass a slice to a function, since it is a pointer to an array you just get a copy of the slice structure. It will still point to the same underlying array segment. So any modifications made to the slice within the function will be seen outside.

https://play.golang.org/p/OLkoxGf8uaZ

However, if you append an element, remember that a new slice is created and elements are copied over so you will lose the elements if this happens within a function so you must return a slice. E.g. append from the stdlib returns a new slice.

Whatever it is you choose to use, stay consistent. Coming from a C++ background, I always thought pass by reference is cheaper than passing by value. But apparently it is not always the case in Go.

Interesting Reads:

http://goinbigdata.com/golang-pass-by-pointer-vs-pass-by-value/