We rolled out this class into QA, which works on various platforms (AIX, HP-UX, Solaris, Linux and Windows). Everything seemed to be okay, unit tests passed and the production code didn’t crash. Alas, QA machines controlled by Hudson/Jenkins are usually overloaded, and quite often it causes very unexpected issues. After a week we discovered that sometimes the code crashed on AIX when calling std::system(). Even more, it crashed inside this function, that was for sure. In most cases any “strange” or “magical” behaviour is related to memory problems. truss, injected into command lines passed to system(), showed that some environment variables in the child process had corrupted values.

We began investigating the new class, EnvironmentVariablesManager. Below is its simplified version but still containing a tricky bug. You may try finding the issue by yourself first, and then read further down.

The simple main() function below easily reproduces the issue.

#include <vector>
#include <map>
#include <string>

#include <unistd.h>

class EnvironmentVariablesManager {
 public:
  typedef std::vector<char> VariableContainer;
  typedef std::map<std::string, VariableContainer> Variables;
  void put(const std::string& name, const std::string& value) {
    VariableContainer pair;
    PairToContainer(name, value, &pair);
    const std::pair<Variables::iterator, bool> inserted =
      vars_.insert(std::make_pair(name, pair));
    if (!inserted.second)
      inserted.first->second = pair;
    putenv(&inserted.first->second[0]);
  }

 private:
  void PairToContainer(const std::string& name, const std::string& value,
                       VariableContainer* pair) const {
    pair->clear();
    std::copy(name.begin(), name.end(), std::back_inserter(*pair));
    pair->push_back('=');
    std::copy(value.begin(), value.end(), std::back_inserter(*pair));
    pair->push_back('\0');
  }
  Variables vars_;
};

int main() {
  EnvironmentVariablesManager env;
  env.put("DB2_HOME", "a");
  env.put("DB2_HOME", "12345678");
}

Valgrind complains on this code saying that putenv tries reading some memory after freeing (this trace is from OSX).

clang++ -o putenv_check putenv_test.cpp && valgrind ./putenv_check
==1046== Memcheck, a memory error detector
==1046== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==1046== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==1046== Command: ./putenv_check
==1046==
--1046-- ./putenv_check:
--1046-- dSYM directory is missing; consider using --dsymutil=yes
==1046== Invalid read of size 1
==1046==    at 0x2A8A3B: __findenv (in /usr/lib/system/libsystem_c.dylib)
==1046==    by 0x232C62: __setenv (in /usr/lib/system/libsystem_c.dylib)
==1046==    by 0x216A7E: putenv (in /usr/lib/system/libsystem_c.dylib)
==1046==    by 0x100001999: EnvironmentVariablesManager::put(std::string const&, std::string const&) (in ./putenv_check)
==1046==    by 0x1000015DE: main (in ./putenv_check)
==1046==  Address 0x100012560 is 0 bytes inside a block of size 11 free'd
==1046==    at 0x563A: free (in /usr/local/Cellar/valgrind/3.8.1/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==1046==    by 0x10000208C: __gnu_cxx::new_allocator<char>::deallocate(char*, unsigned long) (in ./putenv_check)
==1046==    by 0x10000201D: std::_Vector_base<char, std::allocator<char> >::_M_deallocate(char*, unsigned long) (in ./putenv_check)
==1046==    by 0x100002483: std::vector<char, std::allocator<char> >::operator=(std::vector<char, std::allocator<char> > const&) (in ./putenv_check)
==1046==    by 0x1000018A8: EnvironmentVariablesManager::put(std::string const&, std::string const&) (in ./putenv_check)
==1046==    by 0x1000015DE: main (in ./putenv_check)
==1046==
==1046==
==1046== HEAP SUMMARY:
==1046==     in use at exit: 2,425 bytes in 34 blocks
==1046==   total heap usage: 58 allocs, 24 frees, 2,824 bytes allocated
==1046==
==1046== LEAK SUMMARY:
==1046==    definitely lost: 18 bytes in 1 blocks
==1046==    indirectly lost: 0 bytes in 0 blocks
==1046==      possibly lost: 0 bytes in 0 blocks
==1046==    still reachable: 2,407 bytes in 33 blocks
==1046==         suppressed: 0 bytes in 0 blocks
==1046== Rerun with --leak-check=full to see details of leaked memory
==1046==
==1046== For counts of detected and suppressed errors, rerun with: -v
==1046== ERROR SUMMARY: 9 errors from 1 contexts (suppressed: 1 from 1)

To me it was not that obvious what was wrong. For example, if to remove the second call env.put("DB2_HOME", "12345678"), the issue disappeared (valgrind didn’t complain any more). So, we started to suspect the following lines of code:

    if (!inserted.second)
      inserted.first->second = pair;
    putenv(&inserted.first->second[0]);

If to change the code little bit (if fact, doing exactly the same operation but in a slightly different way):

    if (!inserted.second)
      inserted.first->second.assign(pair.begin(), pair.end());

the error report changes:

--1087-- ./putenv_check:
--1087-- dSYM directory is missing; consider using --dsymutil=yes
==1087== Invalid read of size 1
==1087==    at 0x2A8A3B: __findenv (in /usr/lib/system/libsystem_c.dylib)
==1087==    by 0x232C62: __setenv (in /usr/lib/system/libsystem_c.dylib)
==1087==    by 0x216A7E: putenv (in /usr/lib/system/libsystem_c.dylib)
==1087==    by 0x1000016A9: EnvironmentVariablesManager::put(std::string const&, std::string const&) (in ./putenv_check)
==1087==    by 0x10000129E: main (in ./putenv_check)
==1087==  Address 0x100013560 is 0 bytes inside a block of size 11 free'd
==1087==    at 0x563A: free (in /usr/local/Cellar/valgrind/3.8.1/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==1087==    by 0x100001D9C: __gnu_cxx::new_allocator<char>::deallocate(char*, unsigned long) (in ./putenv_check)
==1087==    by 0x100001D2D: std::_Vector_base<char, std::allocator<char> >::_M_deallocate(char*, unsigned long) (in ./putenv_check)
==1087==    by 0x1000022E9: void std::vector<char, std::allocator<char> >::_M_assign_aux<__gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > > >(__gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > >, __gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > >, std::forward_iterator_tag) (in ./putenv_check)
==1087==    by 0x1000021C4: void std::vector<char, std::allocator<char> >::_M_assign_dispatch<__gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > > >(__gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > >, __gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > >, std::__false_type) (in ./putenv_check)
==1087==    by 0x1000020A4: void std::vector<char, std::allocator<char> >::assign<__gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > > >(__gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > >, __gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > >) (in ./putenv_check)
==1087==    by 0x1000015BF: EnvironmentVariablesManager::put(std::string const&, std::string const&) (in ./putenv_check)
==1087==    by 0x10000129E: main (in ./putenv_check)

Now it is more or less clear what is going on. But let’s consider a much more simple example first, purely in C:

#include <unistd.h>
#include <stdlib.h>

int main() {
  char *v1, *v2;
  v1 = malloc(10);
  strcpy(v1, "x=123");
  putenv(v1);
  free(v1);

  v2 = malloc(10);
  strcpy(v2, "x=123");
  putenv(v2);
  free(v2);
  return 0;
}

valgrind also reports about this code (this trace is from Linux):

==523== Memcheck, a memory error detector
==523== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==523== Using Valgrind-3.6.0 and LibVEX; rerun with -h for copyright info
==523== Command: ./putenv_test
==523==
==523== Invalid read of size 1
==523==    at 0x4A07CF9: __GI_strncmp (mc_replace_strmem.c:400)
==523==    by 0x3E1C235649: __add_to_environ (in /lib64/libc-2.12.so)
==523==    by 0x3E1C2353CD: putenv (in /lib64/libc-2.12.so)
==523==    by 0x4A0952D: putenv (mc_replace_strmem.c:1165)
==523==    by 0x400607: main (in /storage2/home3/ademin/test/env/t)
==523==  Address 0x4c28040 is 0 bytes inside a block of size 10 free'd
==523==    at 0x4A0595D: free (vg_replace_malloc.c:366)
==523==    by 0x4005D7: main (in /storage2/home3/ademin/test/env/t)
==523==
==523== Invalid read of size 1
==523==    at 0x4A07D14: __GI_strncmp (mc_replace_strmem.c:400)
==523==    by 0x3E1C235649: __add_to_environ (in /lib64/libc-2.12.so)
==523==    by 0x3E1C2353CD: putenv (in /lib64/libc-2.12.so)
==523==    by 0x4A0952D: putenv (mc_replace_strmem.c:1165)
==523==    by 0x400607: main (in /storage2/home3/ademin/test/env/t)
==523==  Address 0x4c28040 is 0 bytes inside a block of size 10 free'd
==523==    at 0x4A0595D: free (vg_replace_malloc.c:366)
==523==    by 0x4005D7: main (in /storage2/home3/ademin/test/env/t)
==523==
==523== Invalid read of size 1
==523==    at 0x3E1C235652: __add_to_environ (in /lib64/libc-2.12.so)
==523==    by 0x3E1C2353CD: putenv (in /lib64/libc-2.12.so)
==523==    by 0x4A0952D: putenv (mc_replace_strmem.c:1165)
==523==    by 0x400607: main (in /storage2/home3/ademin/test/env/t)
==523==  Address 0x4c28041 is 1 bytes inside a block of size 10 free'd
==523==    at 0x4A0595D: free (vg_replace_malloc.c:366)
==523==    by 0x4005D7: main (in /storage2/home3/ademin/test/env/t)
==523==
==523==
==523== HEAP SUMMARY:
==523==     in use at exit: 0 bytes in 0 blocks
==523==   total heap usage: 3 allocs, 3 frees, 492 bytes allocated
==523==
==523== All heap blocks were freed -- no leaks are possible
==523==
==523== For counts of detected and suppressed errors, rerun with: -v
==523== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 6 from 6)

But if to remove the first free(v1); or move it further down after the second putenv, the valgrind report becomes clean, nothing bad happens.

The conclusion

The putenv function requires that at the moment of setting the new value of a variable the previous value must still exist and be in valid memory (literally it cannot be freed or moved to another location).

For some reason putenv tries reading the previous value when setting the new one.

Now go back to C++. If the original code:

    if (!inserted.second)
      inserted.first->second = pair;

is replaced to:

    if (!inserted.second)
      inserted.first->second.swap(pair);

the issue disappears (valgrind doesn’t find anything suspecious anymore).

Why? The first code destroys (frees, reallocates) the existing value when copying the value of pair into the map. That is why the subsequent call to putenv tries reading a freed memory location.

The second code doing swap only moves the ownership of the data controlled by pair to the element of the map, and in turn the data owned by the element of the map (the existing value) is moved into the pair variable. The pair variable goes out of scope and frees its data only at the end of the function, clearly after calling putenv, so by postponing freeing memory of existing value we allow putenv to read it without any problems.

This is it! End of story.

Frankly, this is the most complicated bug in C++ memory management I came across recently.

Be aware.

I wanted some clarity in this question, so I wrote the following test program:

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#ifdef WINDOWS
#define snprintf _snprintf
#endif
void test(const int capacity) {
  char buf[1024];
  int n;
  strcpy(buf, "abcdefghijk");
  n = snprintf(buf, capacity, "%d", 123);
  printf("capacity=%d, n=%d, buf=[%s] (length %d)\n",
         capacity, n, buf, (int)strlen(buf));
}

int main() {
  test(0);
  test(1);
  test(2);
  test(3);
  test(4);
  test(5);
  return 0;
}

The program tests how snprintf() deals with the buffer when there is no room for the full result, and whether it adds the trailing \0 in this case.

I’ll be testing on different systems and compilers.

Solaris SunOS 5.10 SPARC, Sun C 5.8

capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)

Solaris SunOS 5.10 SPARC, Sun C 5.12

capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)

Solaris SunOS 5.10 Intel x86, Sun C 5.12

capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)

Solaris SunOS 5.11 Intel x86, Sun C 5.12

capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)

Linux 2.6.18 x64, gcc 4.1.2

capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)

Linux 2.6.32 x64, gcc 4.4.6

capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)

HP-UX B.11.31 Itanium 64, HP C/aC++ B3910B A.06.22

capacity=0, n=0, buf=[abcdefghijk] (length 11)
capacity=1, n=-1, buf=[] (length 0)
capacity=2, n=-1, buf=[1] (length 1)
capacity=3, n=-1, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)

AIX 5.3 (PowerPC), IBM XL C/C++ 8.0.0.20

capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)

AIX 6.1 (PowerPC), IBM XL C/C++ 9.0.0.0

capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)

AIX 6.1 (PowerPC), IBM XL C/C++ 9.0.0.15

capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)

AIX 7.1 (PowerPC), IBM XL C/C++ 11.1.0.0

capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)

Windows 7, Visual Studio 12 (17.00.50727.1), x86

capacity=0, n=-1, buf=[abcdefghijk] (length 11)
capacity=1, n=-1, buf=[1bcdefghijk] (length 11)
capacity=2, n=-1, buf=[12cdefghijk] (length 11)
capacity=3, n=3, buf=[123defghijk] (length 11)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)

Windows 7, Visual Studio 12 (17.00.50727.1), x64

capacity=0, n=-1, buf=[abcdefghijk] (length 11)
capacity=1, n=-1, buf=[1bcdefghijk] (length 11)
capacity=2, n=-1, buf=[12cdefghijk] (length 11)
capacity=3, n=3, buf=[123defghijk] (length 11)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)

OSX 10.7.5, Apple clang 4.1

capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)

OSX 10.7.5, Apple gcc/llmv 4.2.1

capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)

The conclusion

On all UNIX systems (SunOS, Linux, AIX, OSX), except HP-UX, the buffer is not touched if it is capacity is 0. Also, the trailing \0 is counted in the length of the output (if the buffer size is 1 it can fit only one character, and this character is the trailing zero). Finally, the function returns the length of data which would be written if the buffer has enough room. This length can be used to allocate another, big enough buffer and call the function once again.

Alas, on HP-UX, if the buffer cannot fit the result, the function returns -1. In this case it is not clear how to figure out the required length of the buffer. By the dichotomy method?

On Windows the situation is even worse. First, the function returns -1 similar to HP-UX if the buffer is not big enough. Second, it does not count the trailing zero in the length of its output, which means it does not append the trailing zero at all in the situation of the small buffer. Though, Microsoft does not recommend using snprintf() at all as a non-thread safe function, and recommends _snprintf_s() instead.

It is obvious now why there are so many “a portable snprintf” implementations in the internet.

Bonus

In my particular case I would be happy using asprintf as an easy to use alternative, but this function is not standard, and, for instance, HP-UX does not support it.

The putenv() function has a quite unpleasant property. It doesn’t take a copy of its argument using it directly by pointer. Moreover, the argument is a non-const pointer. So, it is impossible to pass automatic or temporary objects, and even passing constant strings (which are persistent by default) we need to cast them to non-const ones, which is not quite right. All of this encourages such nonsense like malloc or strdup.

So, there is a class below called EnvironmentVariablesManager. This is a simple wrapper around putenv() and getenv() proving persistent storage for the values passed to putenv() (in the form of “name=value”) .

The class is designed to work on Linux, AIX, HP-UX, Solaris and Windows.

EnvironmentVariablesManager.h file:

#ifndef ENVIRONMENT_VARIABLE_MANAGER_H
#define ENVIRONMENT_VARIABLE_MANAGER_H

#include <string>
#include <vector>
#include <map>

class EnvironmentVariablesManager {
 public:
  typedef std::vector<char> VariableContainer;
  typedef std::map<std::string, VariableContainer> Variables;

  EnvironmentVariablesManager() {}

  void put(const std::string& name, const std::string& value);
  std::string get(const std::string& name) const;
  void del(const std::string& name);

  static void PutOSVariable(char* value);
  static std::string GetOSVariable(const char* name);
  static bool IsOSVariableSet(const char* name);

 private:
  VariableContainer PairToContainer(const std::string& name,
                                    const std::string& value) const;
  Variables vars_;

  // This class is not copiable.
  EnvironmentVariablesManager(const EnvironmentVariablesManager&);
  void operator=(const EnvironmentVariablesManager&);
};

#endif

EnvironmentVariablesManager.cpp file:

#include "EnvironmentVariablesManager.h"

#ifdef WINDOWS
#include <windows.h>
#else
#include <unistd.h>
#endif

#include <vector>
#include <map>
#include <string>
#include <iterator>
#include <algorithm>

#include <cassert>

void EnvironmentVariablesManager::put(const std::string& name,
                                      const std::string& value) {
  const VariableContainer pair = PairToContainer(name, value);
  const std::pair<Variables::iterator, bool> inserted =
    vars_.insert(std::make_pair(name, pair));
  if (!inserted.second)
    inserted.first->second = pair;
  char* data = &(inserted.first->second[0]);
  PutOSVariable(data);
}

std::string EnvironmentVariablesManager::get(const std::string& name) const {
  return GetOSVariable(name.c_str());
}

void EnvironmentVariablesManager::del(const std::string& name) {
  put(name, "");
}

void EnvironmentVariablesManager::PutOSVariable(char* value) {
  ::putenv(value);
}

std::string EnvironmentVariablesManager::GetOSVariable(const char* name) {
#ifdef WINDOWS
  size_t sz = 0;
  assert(getenv_s(&sz, NULL, 0, name) == 0);
  if (sz == 0) return std::string();
  std::vector<char> value(sz + 1);
  assert(getenv_s(&sz, &value[0], sz, name) == 0);
  return std::string(&value[0], sz - 1);
#else
  const char* const value = std::getenv(name);
  return value ? value : "";
#endif
}

bool EnvironmentVariablesManager::IsOSVariableSet(const char* name) {
#ifdef WINDOWS
  size_t sz = 0;
  assert(getenv_s(&sz, NULL, 0, name) == 0);
  return sz > 0;
#else
  const char* value = std::getenv(name);
  return value != NULL && *value != '\0';
#endif
}

EnvironmentVariablesManager::VariableContainer 
EnvironmentVariablesManager::PairToContainer(const std::string& name,
                                             const std::string& value) const {
  VariableContainer pair;                                            
  std::copy(name.begin(), name.end(), std::back_inserter(pair));
  pair.push_back('=');
  std::copy(value.begin(), value.end(), std::back_inserter(pair));
  pair.push_back('\0');
  return pair;
}

Unit-tests using std::assert.

EnvironmentVariablesManager_unittest.cpp file:

#include <iostream>
#include <string>
#include <vector>
#include <cstdlib>
#include <cstring>
#include <cassert>

#ifdef WINDOWS
#include <windows.h>
#else
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#endif

#include "EnvironmentVariablesManager.h"

void Test_EnvironmentVariablesManager_get_put() {
  EnvironmentVariablesManager env;
  assert(std::string("") == env.get("_a_unique_variable_"));
  env.put("_a_unique_variable_", "b");
  assert(std::string("b") == env.get("_a_unique_variable_"));
  env.put("_a_unique_variable_", "abc");
  assert(std::string("abc") == env.get("_a_unique_variable_"));
  env.put("_a_unique_variable_", "");
  assert(std::string("") == env.get("_a_unique_variable_"));
}

namespace {
std::string ReadEnvironmentVariableViaShell(const std::string& name) {
#ifdef WINDOWS
  const std::string shell =
    EnvironmentVariablesManager::GetOSVariable("ComSpec");
  assert(!shell.empty());
  const std::string cmd = shell + " /c echo %" + name + "%";
  FILE* const f = _popen(cmd.c_str(), "rb");
#else
  const std::string cmd = "echo $" + name;
  FILE* const f = popen(cmd.c_str(), "r");
#endif
  assert(f != NULL);
  std::vector<char> line(1024, 0);
  size_t read = 0;
  while (!::feof(f) && read < line.size()) {
    const size_t sz = ::fread(&line[read], 1, line.size() - read, f);
    read += sz;
  }
#ifdef WINDOWS
  ::_pclose(f);
#else
  ::pclose(f);
#endif
  line.resize(read);
  std::string trimmed(read, '\0');
  std::copy(line.begin(), line.end(), trimmed.begin());
  return trimmed.substr(0, trimmed.find_last_not_of("\r\n") + 1);
}
}

void Test_EnvironmentVariablesManager_put_is_propagated_to_child_process() {
  EnvironmentVariablesManager env;
#ifdef WINDOWS
  const std::string empty = "%__unique_%";
#else
  const std::string empty = "";
#endif
  assert(empty == ReadEnvironmentVariableViaShell("__unique_"));
  env.put("__unique_", "b");
  assert(std::string("b") == ReadEnvironmentVariableViaShell("__unique_"));
  env.put("__unique_", "");
  assert(empty == ReadEnvironmentVariableViaShell("__unique_"));
}

void Test_EnvironmentVariablesManager_must_take_a_copy() {
  EnvironmentVariablesManager env;
  char var[] = "12345678";
  env.put("var", var);
  assert(env.get("var") == std::string("12345678"));
  std::strcpy(var, "abc");
  assert(env.get("var") == std::string("12345678"));
}

void Test_EnvironmentVariablesManager_del() {
  EnvironmentVariablesManager env;
  env.put("variable_to_delete", "123");
  assert(std::string("123") == env.get("variable_to_delete"));
  env.del("variable_to_delete");
  assert(env.get("variable_to_delete").empty() == true);
}

void Test_EnvironmentVariablesManager_IsOSVariableSet_set_and_unset() {
  EnvironmentVariablesManager env;
  env.put("a", "value");
  assert(EnvironmentVariablesManager::IsOSVariableSet("a") == true);
  env.put("a", "");
  assert(EnvironmentVariablesManager::IsOSVariableSet("a") == false);
}

void Test_EnvironmentVariablesManager_GetOSVariable() {
  const std::string unique_name = "EnvironmentVariablesManager_GetOSVariable";
  assert(EnvironmentVariablesManager::GetOSVariable(unique_name.c_str())
                                                    .empty());
  const std::string unique_name_pair = unique_name + "=12345678";
  char var[1024];
  unique_name_pair.copy(var, sizeof(var));
  var[unique_name_pair.length()] = '\0';
  ::putenv(var);
  assert(EnvironmentVariablesManager::GetOSVariable(unique_name.c_str())
                                                    == "12345678");
}

void Test_EnvironmentVariablesManager_PutOSVariable() {
  const std::string unique_name = "EnvironmentVariablesManager_PutOSVariable";
  const char* before = ::getenv(unique_name.c_str());
  if (before != NULL)
    assert(std::string(before).empty());

  const std::string unique_name_pair = unique_name + "=12345678";
  char var[1024];
  unique_name_pair.copy(var, sizeof(var));
  var[unique_name_pair.length()] = '\0';

  EnvironmentVariablesManager::PutOSVariable(var);
  const char* after = ::getenv(unique_name.c_str());
  assert(after != NULL);
  assert(std::string(after) == "12345678");
}

int run_tests(int argc, const char* const argv[]) {
  Test_EnvironmentVariablesManager_GetOSVariable();
  Test_EnvironmentVariablesManager_PutOSVariable();
  Test_EnvironmentVariablesManager_get_put();
  Test_EnvironmentVariablesManager_put_is_propagated_to_child_process();
  Test_EnvironmentVariablesManager_must_take_a_copy();
  Test_EnvironmentVariablesManager_del();
  Test_EnvironmentVariablesManager_IsOSVariableSet_set_and_unset();
  return 0;
}

int main(int argc, const char* const argv[]) {
  run_tests(argc, argv);
  std::cout << "All tests pass." << std::endl;
}

Overall, nothing really intricate but quite handy.

My first POS was VeriFone OMNI-395. It was based on Zilog Z180, up to 1M non-volatile memory (for transaction logs, for instance), a Hayes compatible modem (up to 2400BPS), 12 volts RS232 ports for a PIN pad, a serial printer and other devices (alas, not all RS232 ports had all standard line causing inventing a handmade wheel for the flow control.), and an LCD-screen with loadable fonts.

The architecture was pretty interesting. The user code was executed inside a VM running on top of native Z180 code. It allowed dealing with larger amounts of memory which Z180 can address directly, and implementing dynamically loadable user modules (R-modules). Unfortunately it affected overall performance, and, for example, even table-driven CRC16 on 5-10KB of data took a few seconds. Also, the stack size in the C language was quite limited, and sometimes it was worth implementing, for example, the “sprintf” function manually to avoid random crashes due to stack overflows.

But, frankly speaking, comparing to terminal from other vendors back those days (Injenico, Nurit, etc.) where you had to deal with using raw memory blocks, switching pages manually, etc., VeriFone (TXO) provided almost standard C library. In TXO C you dealt with persistent files, serial ports, LCD and other peripherals with read/write/ioctl functions.

The TXO C compiler supported support also loadable modules (R-modules). Such modules can be loaded on the fly from the user code (similar to overlays or CHAIN in the classic BASIC).

A few days ago The Frost Father presented me an old good OMNI-395. Do you want to see it inside? Indeed!

Turn it on, and, (surprise!) it still runs my firmware.

Inside.

The top board, under the keypad, and the bottom one, with connectors.

If I find the compiler and loader I’ll try to write a little demo.

Alas, there is no official documentation about OMNI-395 hardware, but the programmer manuals only cover the standard library and the VM a little bit.

Of course, OMNI-395 can be used as a generic controller. It can communicate via the standard RS232, plus DTR/RTS can be digital output and CTS/DSR – inputs.

Ideally, it is possible to disassemble the ROM (only 64K of Z180 code), and then reverse engineer the architecture. It will allow programming OMNI-395 directly in Z180 assembly.

P.S.

Despite of changing the hardware platform a few times after OMNI-395, VeriFone also provides very good backward compatibility in the programmer’s API. So, porting to newer models is quite simple. After OMNI-395 I ported our POS software to OMNI-3350, 3750, VX510, VX610.

An finally, by the #cardpayments tag at Twitter I post pictures of POS, PIN pads, ATM and other plastic card payment machines I come across. Please, join.

The maximum window size is 100x50 with 64 colours. It supports up to 9 windows simultaneously. Each window is controlled independently via special escape (^[) sequences.

In general – very useful for small projects on microcontrollers if you don’t bother doing VGA output by yourself (RS-232 output can be implemented even manually if your microcontroller doesn’t support it for some reason). I also tested Serial VGA with Raspberry Pi. The only recommendation is to support the CTS signal line on baud rates faster than 9600. Otherwise the device can loose some characters.

The Serial VGA datasheet.

At the moment I have 20 processors (from Intel, AMD, National Semiconductor, NEC, Samsung, Texas Instruments, and also manufactured in the Soviet Union and Czechoslovakia). Amongst chips having a year on the label the earliest is dated back to 1974 and the latest is 1980. All CPUs except one are fully functional. I tested them on my Радио-86РК.

Testing revealed that all processors are identical according to the CPU Exerciser except clones from AMD. The AMD processors, AM8080 and AM9080A, behave differently performing the bitwise AND operation (ANA and ANI instruction). The original Intel CPUs and non-AMD clones set the AC (half-carry) flag to the value of the 3rd bit (A3) from the bitwise OR between the accumulator and the argument of ANA or ANI. The AMD clones always zero the AC flag in the ANA and ANI instructions. I don’t know why the original Intel CPU calculates the AC flag in such a weird way.

My i8080 chips

The double click on the pictures flips the top and the bottom views.

AMD

Interestingly, AMD i8080-compatible chips were reverse-engineered from schematics literally stolen from Intel. So, the Intel vs AMD war began from 8080.

8080A, 1977

AM9080, 1977

National Semiconductor

INS8080AN

This chip is faulty.

INS8080AN

Intel

1974

1977

1979

1980

Russia

KR580IK80A (КР580ИК80A)

KR580VM80A (КР580ВМ80A)

There is the original mini-datasheet in Russian: page 1 and page 2.

NEC

Samsung

Telsa (Czechoslovakia)

Texas Instruments

This is how I took pictures of all these chips in a quite technological way using one iPhone and two Raspberry Pi:

P.S.

I hope the collection will grow. I still have only one chip in white ceramic packaging (KR580VM80). If you know i8080 clones from other manufacturers, which are not listed here, I’d appreciate if you let me know.

If you would like to donate any i8080 chip I’ll be happy putting a reference to you next to the image of the chip.

And you know what, after joining the “dark side” army of Apple users, particularly with Macbook Air, I become much more tolerant paying for software. And the main reason is, you guess?, the easiness of a purchase. With AppStore you just “click-click”, and an application jumps into your Applications in a few seconds. Purchasing directly on web sites is usually simple and smooth as well, especially if you pay using PayPal or Google Wallet.

Moving even further, I listed some apps which I am happy to pay for.

• MacKeeper. In my view this is the number one app for OSX.
• iStat. This is the “must have” number two.
• Sublime. You are a programmer and looking for a text editor? This is the option you won’t regret about.
• Hopper Disassembler. By nature I’m a lower level hacker, so this is the tool I need. I had fallen in love with this app because of the unprecedented feature/price ratio comparing to IDA, its “big brother”.

Now a couple of apps which I also pay for but only because there are no adequate alternatives. I’m more or less happy with functionality but I don’t like their pricing (in short – too much in my view).

• Parallels. Of course, I use VirtualBox as well, but Parallels is so bloody good on OSX.
• ORFO. I’m Russian and always will, so I need a spell and grammar checker integrated into the OS, and ORFO does it pretty well.

However, there is an app, which I’m not willing to pay. Its price is ridiculous. This is the Microsoft Office. Fortunately, the company I work for provided me a license for this product, so I can work with working documents on Mac. When it gets stopped I’ll switch to the Apple alternatives (Pages, Numbers and Keynotes).

And finally, there is an app called Radare. It is free and opensource, and I donated some amount to support this great project (I hope not last time).

P.S.

Of course, there are tons of other free apps (iTerm, 7z, muCommander, Skype, VirtualBox, GitHub, DOSBox, KeePassX, etc.), which I also use but this is a different story.

Agreed, the game is trivial, but I just liked it. Also, I tried ascii.io recording an asciicast, so you can evaluate gameplay - http://ascii.io/a/1715.

The source is available at GitHub’е, but you can simply grab it below.

// Originally taken from http://itblogs.org/c-konsolnaya-zmejka/.

#include <iostream>
#include <cstdio>
#include <cstdlib>
#include <ctime>
#ifdef WINDOWS
#include <windows.h>
#include <conio.h>
#else
#include <unistd.h>
#include <termios.h>
#include <sys/select.h>

#define STDIN_FILENO 0
#define NB_DISABLE 0
#define NB_ENABLE 1

#define Sleep(x) usleep(x*1000)

int kbhit() {
  struct timeval tv;
  fd_set fds;
  tv.tv_sec = 0;
  tv.tv_usec = 0;
  FD_ZERO(&fds);
  FD_SET(STDIN_FILENO, &fds);
  select(STDIN_FILENO+1, &fds, NULL, NULL, &tv);
  return FD_ISSET(STDIN_FILENO, &fds);
}

void nonblock(int state) {
  struct termios ttystate;

  // Get the terminal state.
  tcgetattr(STDIN_FILENO, &ttystate);

  if (state == NB_ENABLE) {
    // Turn off canonical mode.
    ttystate.c_lflag &= ~ICANON;
    // Minimum of number input read.
    ttystate.c_cc[VMIN] = 1;
  } else if (state == NB_DISABLE) {
    // Turn on canonical mode.
    ttystate.c_lflag |= ICANON;
  }
  // Set the terminal attributes.
  tcsetattr(STDIN_FILENO, TCSANOW, &ttystate);
}

int getch() {
  return fgetc(stdin);
}

#endif

int snake_size, change_x, change_y, coordinates_x[1000], coordinates_y[1000];
int food_x = -1, food_y = -1;

char symbol, a[1000][1000];

const int N = 13, M = 17, INTERVAL = 200;

void change_direction() {
  symbol = getch();
  switch (symbol) {
    case 'w': if (change_x != 1 || change_y != 0) {
                change_x = -1; change_y = 0;
              }
              break;
    case 'a': if (change_x != 0 || change_y != 1) {
                change_x = 0; change_y = -1;
              }
              break;
    case 's': if (change_x != -1 || change_y != 0) {
                change_x = 1; change_y = 0;
              }
              break;
    case 'd': if (change_x != 0 || change_y != -1) {
                change_x = 0; change_y = 1;
              }
              break;
#ifndef WINDOWS
    case 'q': nonblock(NB_DISABLE); std::exit(0);
#endif
    default: break;
  }
}

void show_table() {
#ifdef WINDOWS
  system("cls");
#else
  system("clear");
#endif
  for (int i = 0; i <= N + 1; ++i)
    for (int j = 0; j <= M + 1; ++j)
      std::cout
        << (i == 0 || j == 0 || i == N + 1 || j == M + 1 ?
           '#' : a[i][j])
        << (j <= M ? "" : "\n");
}

void clear_snake_on_table() {
  for (int i = 1; i <= snake_size; ++i)
    a[coordinates_x[i]][coordinates_y[i]] = ' ';
}

void show_snake_on_table() {
  if (change_x == 1 && change_y == 0)
    a[coordinates_x[1]][coordinates_y[1]] = 'v';
  if (change_x == -1 && change_y == 0)
    a[coordinates_x[1]][coordinates_y[1]] = '^';
  if (change_x == 0 && change_y == 1)
    a[coordinates_x[1]][coordinates_y[1]] = '>';
  if (change_x == 0 && change_y == -1)
    a[coordinates_x[1]][coordinates_y[1]] = '<';

  for (int i = 2; i <= snake_size; ++i)
    a[coordinates_x[i]][coordinates_y[i]] = '@';
}

bool game_over() {
  for (int i = 2; i <= snake_size; ++i)
    if (coordinates_x[1] == coordinates_x[i] &&
        coordinates_y[1] == coordinates_y[i])
      return true;
  return false;
}

void check_coordinates() {
  if (coordinates_x[1] > N) coordinates_x[1] = 1;
  if (coordinates_x[1] < 1) coordinates_x[1] = N;
  if (coordinates_y[1] > M) coordinates_y[1] = 1;
  if (coordinates_y[1] < 1) coordinates_y[1] = M;
}

void next_step() {
  clear_snake_on_table();

  for (int i = snake_size; i >= 2; --i) {
    coordinates_x[i] = coordinates_x[i - 1];
    coordinates_y[i] = coordinates_y[i - 1];
  }

  coordinates_x[1] += change_x;
  coordinates_y[1] += change_y;

  check_coordinates();

  if (coordinates_x[1] == food_x && coordinates_y[1] == food_y) {
    snake_size++;
    food_x = -1;
    food_y = -1;
  }

  show_snake_on_table();

  if (game_over()) {
    std::cout << "You're looser! \n";
#ifdef WINDOWS
    system("pause");
#endif
    std::exit(0);
  }
}

bool food_check() {
  if (food_x == -1 && food_y == -1) return false;
  return true;
}

void place_food() {
  std::srand(std::time(NULL));
  for (int i = 1; i <= 9; ++i) {
    int x = std::rand(), y = std::rand();
    if (x < 0) x *= -1;
    if (y < 0) y *= -1;
    x %= (N + 1);
    y %= (M + 1);
    if (x == 0) ++x;
    if (y == 0) ++y;
    if (a[x][y] != '@' && a[x][y] != 'v' && a[x][y] != '^' &&
        a[x][y] != '<' && a[x][y] != '>') {
      food_x = x;
      food_y = y;
      a[x][y] = '+';
      return;
    }
  }
}

void standart_settings() {
  snake_size = 2;

  coordinates_x[1] = 1;
  coordinates_y[1] = 2;
  coordinates_x[2] = 1;
  coordinates_y[2] = 1;

  change_x = 0;
  change_y = 1;
}

int main() {
  standart_settings();

#ifndef WINDOWS
  std::memset(a, ' ', sizeof(a));
  nonblock(NB_ENABLE);
#endif

  while (true) {
    if (kbhit() != 0)
     change_direction();

    next_step();

    if (!food_check())
      place_food();

    show_table();

    Sleep(INTERVAL);
  }
}

P.S.

By the way, if you have similar cool console stuff, please share.

Recently Geoff has released a new, updated colour version modification having a few extra features along with the colour (new and improved features are marked with the asterisk):

• PS/2 keyboard
• VGA output with eight colours (480x432 or 240x216) (*)
• monochrome Composite Video output
• synthesised music and sound effects (*)
• battery backed real time clock (*)
• 20 external I/O lines on the back panel
• Arduino compatible connector (*)
• SD card for storing programs and files (up to 32GB)
• USB connectivity as a terminal or for file transfer
• protocols including Serial, I2C, SPI and 1-wire
• dual-channel PWM analog output (*)
• firmware upgrades via USB
• special commands for animated games

MMBasic allows to take full advantage of all these features. It is even possible implementing the PIC32 timer interrupt routines directly in BASIC.

As previously I have ordered the new Maximite Kit from Altronics.

There are no on-surface planar elements which makes the life of soldering laymans like me much easier. There is only one planar capacitor (C10) for some reason. It is quite small, so proper soldering it took me some time to avoid shortening on the board.

All done.

The Original and Colour Maximite.

In general, the kit from Altronics is very high quality. Recommended.

This is what MMBasic allows a user to do with colours.

Well, no colours though, but just classics.

As the author explains, generating colours requires the 100-pin version of PIC32 with three SPI channels. Obviously, it also requires three times much work processing the colour information. Because the crystal still works on the same frequency as previously, 80MHz, the colour Maximite works a bit slower than its black-and-white predecessor. But along with the firmware providing colours, there is a black-and-white version of MMBasic for the new Maximite having all new features except the colour video, so it works with the same speed as the original Maximite.

Well, as previously I have a lot of fun using Maximite, and strongly recommend it if you are up building a nice powerful microcomputer in a few hours.

• a special log forwarder (from Splunk) which can efficiently send log updates to the server
• as regular files via NFS or SMB
• SNMP
• directly over TCP/IP (for example, instead of creating log files in the first place)

On Windows Splunk can take information from Windows Events, Performance Counters or Registry.

For Splunk logs are textual files split into lines. When indexing logs Splunk parses log lines into fields, for example, in the “name=value” format (which is also configurable though). Then using a special query language called SPL it allows manipulating with these fields: sorting, aggregation, computed fields, creating views and tables, external dictionary lookups (for instance, to RDBMS), and, of course, creating various graphic representations of data. Even though normally SPL operates with individual log lines, it supports multi-line queries, which can be useful, for example, to extract transactions or any other activities generating multiple log records.

As Splunk creators state, all logs kept and indexed by Splunk despite of their age are equally available for querying. There is no concept of archiving. Of course, machines running Splunk should correspond to the volume of logs being processed.

Also Splunk states itself as “Google for logs”, but let’s leave it without comments.

The UI of Splunk is web. It allows creating dashboards, which in turn can be presented as a Splunk app. Splunk has its own app store, where most of apps are free though. There are apps already developed, for example, for UNIX syslog, Apache logs, Microsoft Exchange, etc.

Splunk software is freely available on the official website. The licensing model is based on volumes of logs processed daily. Under some small threshold Splunk is free. This threshold is surely enough for evaluation.

I would recommend a book called Exploring Splunk for those who want quickly dig into Splunk and figure out its main features. This book provides very solid overview of SPL and other concepts how Splunk works.

In this article I would like to show a real example of using Splunk. This exercise can be repeated in half an hour. The only prerequisite is to download and install Splunk on your operating system. Then just follow my instructions.

The example is a bit unusual though. Traditionally, logs are used for post factum analysis. But nothing really prevents from picking up updates on the fly and building “live” indicators based on logs. Agreed, my example may look a bit artificial, but I want to show how quickly to feed data into Splunk and formalize it, and finally create a dynamic UI with it.

Let’s begin with a simple script in Ruby. It generates log records containing a number representing its completion (from 0 to 100%).

    require 'date'

    duration = 60*1
    update_period = 0.5
    i = 0
    while i <= duration do
      progress = i * 100.0 / duration
      msg = "%s progress=%05.2f\n" % [DateTime.now, progress]
      puts msg
      open("logs/my.log", 'a') { |f| f << msg }
      i = i + update_period
      sleep update_period
    end

The log records may look like:

2012-11-23T15:58:54+00:00 progress=45.00
2012-11-23T15:58:55+00:00 progress=45.83
2012-11-23T15:58:55+00:00 progress=46.67
2012-11-23T15:58:56+00:00 progress=47.50
2012-11-23T15:58:56+00:00 progress=48.33
2012-11-23T15:58:57+00:00 progress=49.17
2012-11-23T15:58:57+00:00 progress=50.00

We will develop a dashboard in Splunk, which displays the percentage in the form of a cool indicator taking updates from logs.

For simplicity we run everything on a single machine, and Splunk will be taking log updates directly from a file.

So, you have already installed Splunk, and if you open “http://localhost:8080” in your browser you should be able to login as the “admin” user.

Then in menu follow to: “Manager -> Data Inputs -> Add data -> A file or directory or files”. Here we specify the file and directory name with the logs (in our case just a single file).

Confirm creating a source of logs (source type). As I said, logs can be delivered to Splunk in various ways. Each way can be named and addressed as a source type.

The log has been added. We see that Splunk has picked up the file and already parsed its lines into fields. Out of the box Splunk understands many date and time formats, but it is configurable.

Now we give a name to our source type - “test_logging” and save the settings.

Now we get back to the main page and type in our first SPL query in the search bar (in red):

sourcetype="test_logging" | table progress as float

It says: take logs from the “test_logging” data source, create a table with one column called “progress” taking values from the corresponding field, set the type of “progress” to float. At the bottom (in blue) we see the result of the query (own log already has some data). SPL uses the concept of UNIX pipes (|) feeding output from one command to the next one.

So, we have the table. Now let’s create a widget. Due to simplicity of our data (one single field) we can represent it as a speedometer gauge. Click on “Formatting options” (in blue) and set the chart type to “radial gauge” (in red). And here is our cool widget:

The first widget is ready. Now let’s create an alternative one looking differently. It will also represent the value of the progress but in the form of a horizontal bar moving from left to right. This is the query:

sourcetype="test_logging" | table _time progress | head 1

It says: using data from the “test_logging” source type create a table with two fields, “_time” and “progress”, and then take only the first line from it. Default sorting is descending by “_time”. At the bottom (in green) we see the result of the query.

Now click on “Formatting options”, choose “bar” (in green), set the interval for the Y-axis from 0 to 100. For some reason, the X-axis is vertical here (this axis will represent “_time”), and the Y-axis is horizontal (this one will show “progress”). Because our query on the previous picture already shown “100”, the widget is fully filled.

Describing the first widget I deliberately skipped how to save the query. We will do it now. We click on “Create” and then “Dashboard panel…” (in red) and save the settings. We name the first widget as “Speedometer” and the second - “Progress bar”.

Along with saving the first widget we will be asked to create a new panel, a dashboard. Let’s call it “Test logging”. When saving the second widget we will add it to the already created one.

Finally, we click on “Dashboards & View”, select your dashboard and it will be shown like:

There is no data here, so the panels are empty. We see the name of the dashboard (in red), the names of the widgets (in yellow) and two “Edit” buttons (in blue) allowing updating queries and visual representation on the fly. Before starting our Ruby script, we need to click on the both “Edit” buttons and set the time interval from “rt-1s” to “rt”. Such interval means getting updates each second.

Now let’s click on “On” and kick off the script!

A little video how it works:

That’s it.

The example, as I said, is very simple, but I hope it gives you as least initial understanding of Splunk.

Main distinctive features of Propeller:

• 8 independent cores (cogs) working in parallel. Any resource sharing (memory or I/O) is handled by hardware and isn’t under programmer’s control. This guaranties predictable latency. Each cog (core) has its own private 4KB RAM, plus an ability to generate the TV or VGA video signal.
• There is no concept of interrupts in any form. Instead you run concurrent tasks on different cogs.
• Two main programming languages available: a high level one called Spin, and the assembly language. Spin simplifies concurrent and multi-core programming, and its interpreter is build-in to the crystal.
• No almost a concept of programming or flashing the crystal. The higher half of the address space (32KB) is ROM and contains the Spin interpreter and other system tables. The lower half is RAM and it is volatile. It needs to be loaded on each reset. In real applications an external EEPROM memory chip is connected and Propeller copies 32KB from that EEPROM (via I2C) to RAM.
• Propeller is declared as 32-bit because it supports 32-bit operations, but the address space is 16-bit (64KB).

The Spin language was designed to make concurrent and multi-core programming easier. This language is a mixture of procedural and object-oriented approaches.

Below is an example of code in Spin launching a function to spin on three cogs. The code is really easy to read and understand.

CON

  _clkmode = xtal1 + pll16x         'Establish speed
  _xinfreq = 5_000_000              '80Mhz

OBJ

  led: "E555_LEDEngine.spin"        'Include LED methods object

VAR

  byte Counter                      'Establish Counter Variable
  long stack[90]                    'Establish working space

PUB Main

  cognew(Twinkle(16,clkfreq/50), @stack[0])    'start Twinkle cog 1
  cognew(Twinkle(19,clkfreq/150), @stack[30])  'start Twinkle cog 2
  cognew(Twinkle(22,clkfreq/100), @stack[60])  'start Twinkle cog 3

PUB Twinkle(PIN,RATE)                  'Method declaration 

  repeat                               'Initiate a master loop

    repeat Counter from 0 to 100       'Repeat loop Counter
      led.LEDBrightness(Counter, PIN)  'Adjust LED brightness 
      waitcnt(RATE + cnt)              'Wait a moment

    repeat Counter from 100 to 0       'Repeat loop Counter
      led.LEDBrightness(Counter,PIN)   'Adjust LED brightness 
      waitcnt(RATE + cnt)              'Wait a moment

The cognew function kicks off the function Twinkle on three cores and parameterizes each one with a frequency and stack.

The diagram of the Propeller architecture:

The name “Propeller” comes from how the Hub controlling resource sharing switches between cogs. It does round-robin similar to the spinning propeller.

I don’t want to go deeper into Propeller in this post because this is a long story. Instead I put a list of very useful books at the end. They provide the exhaustive details.

But I’d like to share about an interesting project called “The Pocket Mini Computer” based on Propeller (P8X32A). This project uses the “P8X32A QuickStart” evaluation board as a base.

The photo from the official website:

In fact, the author offers the evaluation board, plus the extension, which has VGA, microSD, PS/2, sound and Wii Gameport interfaces. Optionally, the 32KB SRAM chip can be added.

The point of the project is a BASIC interpreter running on this computer. It converts it to a micro-computer a-la from 80s. The interpreter is written in Spin (sources are open). The dialect is quite limited though: no arrays, real and string variables, one letter variable names etc. Nevertheless, BASIC provides access to all peripherals including the SD-card. Plus, it allows running native binaries which can be developed in assembly, C (Parallax has a GCC version for Propeller) or Spin.

Below there are a few pictures of the kit to understand how it looks like. As I said, PMC is based on the P8X32A evaluation board, so only the extension board requires assembly (soldering).

Almost everything is in place.

The sandwich is ready.

A small demo of graphics capabilities.

Overall impression

Propeller

It is not easy to call it “a general purpose microcontroller”. In my subjective view it fits perfectly into certain applications, like, for instance, games consoles or mini-vending machines because it can generate a good quality video signal out of the box. But, on the other hand there are no built-in PWM, ADC/DAC, flash memory, triggers, interrupts, and the creators recommend implementing such functionality manually or using external chips. In the books listed in the end there are plenty examples of such applications.

Conslusion: the very interesting and worth considering architecture.

The PMC kit

This is a very interesting and entertaining project. Of course, it is obvious that BASIC is very limited in resources, for instance, RAM. Maximite, for example, based on PIC32, is much more powerful. It can easily run not only its powerful MMBasic, but, for example, RetroBSD or even emulate Radio-86RK.

But, $39 is the great offer for those who want to play with Propeller having complete hardware.

Dessert

Here is a list of books I read (regarding the architecture) and skimmed (regarding projects). All are recommended.

A short, concise and easy book for beginners. It describes many projects on Propeller. One of the authors is the PMC inventor.

Getting Started With the Propeller

A very solid and easy to read book. It explains in all details the architecture of Propeller. Though the author doesn’t cover programming in assembly, only in Spin, but he precisely shows how to use properly and benefit from the original multi-core architecture of Propeller. The first section covers the internals of Propeller. Following sections contain various projects (can be skipped on the first reading).

Programming the Propeller with Spin : A Beginner’s Guide to Parallel Processing (Tab Electronics)

A collections of real applications with Propeller from its creators.

Programming and Customizing the Multicore Propeller Microcontroller : The Official Guide

A small lyrical digression.

Does any know that Peter Norton didn’t write the famous Norton Commander? The Norton Commander had been invented and then maintained by John Socha up to its “canonical” version 3. Socha used quite an unusual approach in those days mixing the assembly language with C when most of “true” developers programmed purely in assembly. It allowed developing sophisticated but still fast and efficient applications much quicker.

Okay, go back to the book. It liked the idea of buying old books almost for nothing just for the sake of holding them in hands.

Wonderful. They sent me a decommissioned library copy carefully glued by the sticky tape.

I personally don’t like when people write in books or fold page corners, but this looks very touching – somebody did drilled this stuff.

Is it still possible NOT to understand how EQU works?

Or DUP?

Or stack?

Or how to intercept interrupts in DOS?

By the way, it turned that I have got the third edition of the book having a couple of extra chapters about 286 and 386 comparing to its first edition which I read years ago in Russian.

The bottom line – this book was and still is fantastic.

Through the book they explain step by step a process of developing a program called “dskpatch”, a visual editor allowing viewing and modifying individual sectors on floppy or hard drives. In the end it turns into an advanced product split into modules and having dozens of generic routines easily re-usable in any program. Starting from the very beginning you have to learn the basics of 8086, then BIOS and DOS functions, the direct access to video memory and keyboard.

But the main thing is that this book explains how to write really big projects in assembly. For example, my real opening was that if each subroutine always preserves the registers which are not to pass parameters and to return the result the amount of “unpredictable side effects” drops down substantially. Of course, today it sounds like information from a nursery level, but even in the nursery somebody has communicate it.

In the appendix there is the full carefully commented listing of “dskptach”.

Also, the third edition has a few chapters about mixing the assembler and high level languages like C and PASCAL. Passing function parameters to and from the assembler routines is explained in all details. As a bonus, there is a library with an unambiguous name – SOCHALIB congaing a lot of very useful routines for the screen, mouse and the keyboard. It was delightfully impressed that a routine putting a character on the screen used a pre-calculated lookup table to compute an address of the screen line base instead of multiplying Y times 80 directly. I suspect the code of the library resides somewhere inside the Norton Commander and Norton Utilities, and it explains why these products worked blazingly fast.

The conclusion: strongly recommended as an example of the perfect textbook.

How cool is that? Let’s check it out. It has arrived:

The parts:

Let’s begin its “by slice” composing:

After the first two slices let’s put the RPi (remember that it’s design doesn’t have even holes for screws):

Slice by slice:

Now screws and nuts…

The device is ready!

Frankly I just love Raspberry Pi. It is difficult to imagine a better device for schools and universities for studying computers. Only Maximite could compete if we need a bit more lower level, closer to hardware.

Before my RPi’s “case” from SK Pang looked like this:

Incredibly useful device if you need access to GPIO, but as a “normal” case Pibow is unbeatable.

BEGIN
   if (Hi(Code) == 0)
    BEGIN
     BAsmCode[0] = Lo(Code); CodeLen = 1;
     return True;
    END
   else if (MomCPU <= CPU1802)
    BEGIN
     WrError(1500);
     return False;
    END
   else
    BEGIN
     BAsmCode[0] = Hi(Code); BAsmCode[1] = Lo(Code); CodeLen = 2;
     return True;
    END
END

(grabbed from code1802.c).

Frankly, it is not easy to recognize the language. Of course, this is C but initially it was Pascal, and then the author had converted in to C trying to minimize the amount of changes.

The project is called “Macro-assembler AS”, by Alfred Arnold. I use it as an Intel 8080 macro-assembler available on Mac, Linux and Windows. Frankly I haven’t seen anything better so far.

As the author explains, the project was started in Turbo Pascal, but after Borland had abandoned the DOS version but free implementation of the modern Pascal didn’t exist yet, he decided to convert the sources into C and continue development in C. Despite of overall doubtfulness of such approach, to me, the author had managed it very well. The project hadn’t die as it usually happens with big projects after rewrite. But though I have cloned it for myself and started using successfully, I doubt that I want to contribute because it requires coding in this weird “Pascal on C steroids” dialect. Alas, this is inevitable due to that conversion, otherwise the project may turn into mess. If to surf amongst the sources of AS it is easy to find a lot of tricks like “how to make C to be Pascal”.

Nevertheless, all the best to Alfred with his great project. Again, the project moves forward and becomes better.

By the way, Alfred has a very cool collection of old hardware which makes me feel envy.

Introduction to 8080-8085 Assembly Language Programming (Self-teaching Guides)

A reasonable question “why”? Especially considering that in the last couple of months I have re-written from scratch two emulators of the Intel 8080 microprocessor, in C and JavaScript. Do I still have any questions about 8080 assembly? No.

Frankly, I just heard about this book as a perfect example of the assembly textbook. Something similar to the famous Maurer’s book on programming on 6502 assembly for Apple computers. I believe the digitized copies are available because the book was published years ago back in 1981, but I had come across a very cheap hardcopy (~6 pounds including delivery) at Amazon, second-hand, of course.

Have no idea why but when holding such old books in hands I feel some unexplainable excitement from all these hand-made annotations, old-fashion fonts for code and punch card forms.

And in fact the book is great. If, just in some weird and unusual case you decide to learn the 8080 assembly language, but you have never heard about 8080 and the assembly language in general, this book is the perfect option.

It begins from the very beginning – dealing with binary, decimal and hexadecimal calculations; then it follows through groups of instructions with many examples – array indexing, multiplication and division, BCD representation, the concept of stack and so on. Each chapter is concluded with exercises.

The very good book. Recommended.

git diff --shortstat rel-2010 rel-2011

It prints something like this:

12 files changed, 2462 insertions(+), 488 deletions(-)

If your project isn’t in git, you can quickly import the branches you want to analyse in a temporary git repo.

I know that C is THE low-level assembler, and when it is used properly it generates code very close to the assembly. But nevertheless there are some systems still where the C compiler is difficult to implement efficiently. For example, I wanted a C compiler for Intel 8080 to write a program for Radio-86RK. So far, I have found only a couple derivatives of the famous Small-C – smallc-85 and smallc-scc3 which compile and work.

Unfortunately, even for trivial code:

main() {
  static char a;
  for (a = 1; a < 10; ++a) {
     ++a;
  }
}

It generates the following rubbish:

;main() {
main:
;  static char a;
    dseg
?2: ds  1
    cseg
;  for (a = 1; a < 10; ++a) {
    lxi h,?2
    push    h
    lxi h,1
    pop d
    call    ?pchar
?3:
    lxi h,?2
    call    ?gchar
    push    h
    lxi h,10
    pop d
    call    ?lt
    mov a,h
    ora l
    jnz ?5
    jmp ?6
?4:
    lxi h,?2
    push    h
    call    ?gchar
    inx h
    pop d
    call    ?pchar
    jmp ?3
?5:
;     ++a;
    lxi h,?2
    push    h
    call    ?gchar
    inx h
    pop d
    call    ?pchar
;  }
    jmp ?4
?6:
;}
?1:
    ret

Agreed, there are a lot of questions to the compiler, but in general Intel 8080 isn’t quite friendly for the C compiler: no multiplication and division, no indirect addressing on the stack, etc.

Okay, back to Forth. When thinking about Forth for I8080 I wrote a handy macro assembler (it’ll be a separate blog post) and along the way remembered about my old project back to FidoNet times: F-CODE. To obfuscate the code against tracing and disassembling I implemented a micro Forth kernel with direct threading.

“Implemented a micro Forth kernel” sounds “cool” but in reality the direct threading code interpreter is almost trivial:

; F-Code Address Interpreter

GetNext$:       cld
                mov     si, IP$
                lodsw
                mov     IP$, si
                retn

CALLR$:         add     RP$, 2
                mov     bp, RP$
                mov     ax, IP$
                mov     [bp], ax
                pop     word ptr IP$
                next

RETR$:          mov     bp, RP$
                mov     ax, [bp]
                mov     IP$, ax
                sub     RP$, 2
                next

NEXT$:          call    GetNext$
                jmp     ax

osPush$:        call    GetNext$
                push    ax
                next

NEXT            MACRO
                jmp     NEXT$
                ENDM

Plus, a few extra primitives implemented in assembly:

; Adc  ( a b -> c isCarry )
; if a+b>FFFF isCarry = FFFF else isCarry=0

osAdc$:         pop     ax  dx          ; -> a b
                add     ax, dx
                sbb     dx, dx
                push    ax  dx          ; c isCarry ->
                NEXT

; osSwap ( a b -> b a )

osSwap$:        pop      ax bx
                push     ax bx
                NEXT

; osRot ( a b c -> b c a )

osRot$:         pop      ax bx cx
                push     bx ax cx
                NEXT

osPut$:         add     RP$, 2
                mov     bp, RP$
                pop     word ptr [bp]
                NEXT

osGet$:         mov     bp, RP$
                push    word ptr [bp]
                sub     RP$, 2
                NEXT

osDrop$:        add     sp, 2
                NEXT

; osNor ( a b -> a NOR b )

osNor$:         pop     ax bx
                or      ax, bx
                not     ax
                push    ax
                NEXT

osTrap$:        int     3
                NEXT

; osPeek ( addr -> value )

osPeek$:        pop     bx
                push    word ptr [bx]
                NEXT

; osPoke ( Value Addr -> )

osPoke$:        pop     bx              ; -> Value Addr
                pop     word ptr [bx]   ; ->
                NEXT

Now we have a complete stack machine which we can program. Obviously, when tracing or disassembling the threaded code there are only jumps back and fourth making it very difficult for reverse engineering (the purpose of the obfuscation). If you’re curious you may try digging into fcode.com. This little program asks a password which you need to guest/crack/patch/etc. Note that this is a DOS binary, so it needs, for instance, DOSBox to run.

For example, code to compute CRC on this stack machine:

CalcCRC:        CALLR                 ; ->
                ofPush  0             ; CRC
                ofPush  0             ; CRC 0
                ofPeekb Buffer+1      ; CRC 0 Size
                $For                  ; CRC
                    osI                   ; CRC i
                    ofPush  Buffer+2      ; CRC i Buffer+2
                    osAdd                 ; CRC Addr
                    osPeekb               ; CRC Value
                    osExch                ; CRC Value*256
                    $For    0, 8          ; CRC Value
                        osShl                 ; CRC Value*2 isCarry
                        osRot                 ; Value*2 isCarry CRC
                        osSwap                ; Value*2 CRC isCarry
                        osRcl                 ; Value*2 CRC*2 isCarry
                        $If <>0               ; Value*2 CRC*2
                            ofXor 8408h           ; Value*2 CRC*2^Const
                        $Endif
                        osSwap                ; CRC*2 Value*2
                    $Loop                   ; CRC Value*2
                    osDrop                ; CRC
                $Loop                 ; CRC
                RETR

Awesome, right?

When working on F-CODE a simple preprocessor for the assembler language was born allowing to write code like this:

 lea dx, msg2
 cmp bh, 3
 $if <>0
   lea dx, msg1
 $else
   hlt
 $endif

 cmp dx, 0C0DEh
 $if =0
   lea dx, msg2
 $endif

 mov cx, 2
 $Do
   $Do
   cmp ax, 1
   $EndDo =
   dec cx
 $EndDo Loop

 Store ds, si, ax
     $Do
       cmp al, 1
       $if <>0
         $ExitDo
       $endif
       Store ax, bx, cx, es, bp
         ...
       Restore
       $ContDo
     $EndDo Loop
 Restore

Similar to other projects back in DOS times I wrote the preprocessor in Turbo Pascal.

Of course the F-CODE project now has predominantly historical value, although nothing stops to implement the interpreter, for instance, in JavaScript and then re-use existing primitives without any changes.

The F-CODE project is available at GitHub – https://github.com/begoon/fcode.

It requires TASM/TLINK and Turbo Pascal 5.+ to build. Obviously, it also requires DOS to run.

P.S. By the way, people develop quite sophisticated software in Forth. For example, nnbackup is fully written in Forth.

Lighter, thinner, faster.

The new connector, plus the headset is at the bottom now.

The new connector (male).

People are seriously stuck to desks with the iPhone 5.

By they way, the new 3D-maps in iOS6 are fantastic! Unfortunately, not available for my iPhone 4 (not S).

The traditional sound check on the right website.

A little followup.

Regardless that I’m fully on Apple products, I believe that I still have at least a little bit of common sense. If some Apple products are bad or even shit (like Magic Mouse, for instance) I joyfully state that. But brothers! The iPhone 5 is an amazing product of design and engineering. God bless Apple’s designers and engineers. You may hate Apple for their pathos and show off, you may try suing them and even win. But! (Samsung, for example) Please! Create something even close to the iPhone 5. And all Apple’s money and glory will be yours.

Two compression libraries were discovered:

• miniz (zlib and deflate algorithms)
• minilzo (LZO algorithm)

The selection criteria were:

• a single source library without any dependecies
• scrict ANSI C (I used the Microchip XC32 compiler)
• in-memory decompression
• a small static amount of temporary memory for decompression (no malloc/calloc) (ideally, it should use the output decompression buffer only)

The both libraries compiled and worked for PIC32 without any issues.

On my data miniz provided the 0.78 compression ratio and minilzo - 0.71. So, both didn’t squeeze my data into the 512KB flash of PIC32.

Overall impression

• miniz is slightly easier to use and more powerfull from the API perspective
• minilzo provides better compression

Also miniz uses only the output buffer for decompression, but minilzo requries at least 16K static buffer.

P.S. I also tried XZ Embedded (LZMA2). It looked that it compressed much better but it requires malloc/free API, so I didn’t manage to build it on PIC32 without extra development.

I wanted a simple and quick app helping me to remember a list of drink in the round. Plus, I wanted to write something for iOS in general. Eventually, I ended up with this:

Quick and dirty, but it solves the problem in 100%. I personally use this app all the time. Unfortunately, AppStore has rejected it as the app having too limited functionality. Well, bad for them, good for me. I have learned a lot developing this app.

I had to learn the following classes:

• UITabBarController to manage two tab views.
• UIViewController to work with tables via UITableViewDataSource and UITableViewDelegate delegates.
• Object serialization via NSKeyArchiver to save the list of drink to a file.

The full project is available at GitHub - https://github.com/begoon/buyround. Please, feel free to checkout and build it.

As I wrote previously about my first iPhone app - US Visa, the quality of the artwork, especially, icons, plays the crucial role in the success of a mobile application. I’ve googled one (below) but it is much better to design your own one.

P.S. For the sake of making this post complete I’ve decided to attach a photo of absolutely inhuman conditions of creating it.

There is a DOS version, but, unfortunately, the gameplay was slightly changed. In the Apple’s version the opponents inside the house never approach you first. There was a trick: a little step towards and immediately step back. This was the only way to force them to attack you. Then you can low-kick them. In the DOS version they always attack first, so just low-kick.

But for the DOS version there is a magic patch allowing to speedrun the game.

Immortality
KARATEKA.EXE
00003066: 48 90
00003D7E: 48 90

Steel breast
KARATEKA.EXE
0000306E: 83 C6
0000306F: 3E 06
00003072: 3F 01
00003073: 7E 75

Flying kick for humans
KARATEKA.EXE
00002F30: 7E 00

Flying kick for bird
KARATEKA.EXE
00002E2F: 7F 00
00002E30: 01 00
00002E34: 3D 25
00002E35: 04 00
00002E44: 06 00

Kill bird by first kick
KARATEKA.EXE
000031BA: 85 33

Kill humans by first kick
KARATEKA.EXE
00002F3A: 85 33

For example:

Don’t miss at the end when the princess hopelessly kicks you when you approach in the stance.

P.S. Unfortunately, the sources of this spectacular game are still closed.

Its DOS implementation is a quite precise replica perfectly working on modern systems in DOSBox:

dosbox -exit BOLO.COM

Years ago I created a patch to know what happens at the end:

Unlimited lives
BOLO.COM
00000174: 4F 00
00000178: DF 4A
00000179: FE FF

Unlimited fuel
BOLO.COM
00000A38: 06 04

Immortality & pass through the walls
BOLO.COM
00000C61: 01 00
0000172D: 01 00
00001817: 01 00

With this patch you can play like this:

Unfortunately, there are no sources of the original BOLO available, so porting to SDL, for instance, is fully about reverse engineering of the original binaries (either of the Apple II version or the DOS clone), which is not that easy. I know there are some other BOLO clones for Windows are available, but they are physically different games, not original BOLO.

It costs 128€ + 25.80€ VAT for the 8 channel model. They have stock in the UK, so delivery was free, fast and no customs involved.

At the beginning I had a cunning plan to buy, play for a while and then return. They provide an option for full money back within a month without explaining anything. This is very good. Frankly, if there was no such option, I would never buy it.

It turned out to be a match box size thing in the solid metal case.

The complete set.

Connecting…

To begin with I connected it to a serial port on Raspberry Pi, which I was using as a telnet console.

Here is a screen shot of the capture. We see that the ls command is flying to one direction and being echoed back from the second one.

Now the GMC-4 keyboard. This is a scanned keyboard where one (out) port is used to set a “flying” pin selecting the current row (or column) of buttons, and the second (in) port represents the state of buttons in the selected row.

I planned to simulate the keyboard, so I needed timings of the scanning signal. I connected all four scanning lines to four channels, and got the following:

Now it is crystal clear what’s going on. It allows to measure a variety of parameters of the signal, scale it and compare against previous runs. There is a good quality video available on the official website showing some use cases in details. My status in electronics is a layman, so I was simply impressed by the ease of use. The guy, Viacheslav, who recommended me this device, says that for debugging a complicated digital circuit design it is crucial to be able to capture much more than one or two channels simultaneously.

As you have probably realised, I didn’t return the device, but I had a complaint though. I wasn’t able to capture on the highest frequency of 24MHz. The Logic software complained about lack of USB bandwidth when transferring data from the Logic to the computer. The Support suggested an issue with my particular USB hub, but I tried different setups and the problem still stands. But they were ready to take the device back and refund anytime. Regardless, I liked the device and kept it. It perfectly solves all my current needs.

The conclusion

Pros

• Extremely simple is use.
• Configurable triggers to initiate capturing (by level, by front, etc.).
• 8 or 16 channels depending on the model for simultaneous capture.
• 100% money back within a month if you’re uncertain.

Cons

• No online mode. You capture first, then view, analyse etc. So, don’t expect “regular oscilloscope experience”.
• My Mac Air wasn’t able to capture on the maximal frequence of 24MHz.

One liner

Recommended.

It’s cool but quite useless. There is another interesting feature. E-mails sent to the addresses below:

• comrade+fred@gmail.com
• comrade+john@gmail.com
• comrade+peter@gmail.com …
• comrade+a_somebodies_name@gmail.com

will be also delivered to comrade@gmail.com. Then it is possible to set some filtering rules in Gmail redirecting these e-mails to appropriate people depending on that extra bit in the “From:” address between + and @ symbols.

This may help organizing a free “pseudo” mail server with Gmail.

By the way, there is no prize for solving this problem except the honour and a link which I can put with the solver’s name.

Please, don’t post the solution anywhere in comments letting others to try solving it independently.

That was the first, simplest level, but how about this one?

The game perfectly works in DOSEmu: DOSEmu -exit pusher.exe.

Lazy people can checkout a quick video instead:

I was curious how 60 maps fitted into such a tiny executable. After a bit of fiddling with IDA I wrote a simple program to extract the maps from the pusher.exe binary and print them out. For example:

*************************************
Maze: 1
File offset: 148C, DS:00FC, table offset: 0000
Size X: 22
Size Y: 11
End: 14BD
Length: 50

    XXXXX
    X   X
    X*  X
  XXX  *XXX
  X  *  * X
XXX X XXX X     XXXXXX
X   X XXX XXXXXXX  ..X
X *  *             ..X
XXXXX XXXX X@XXXX  ..X
    X      XXX  XXXXXX
    XXXXXXXX

*************************************

All the map are available.

The maps are compressed with a Huffman-like approach using bit sequences of variable length. Each level has the following structure:

• map size X (1 byte)
• map size Y (1 byte)
• X*Y bytes of the map represented as a sequence of pairs <COUNTER><CODE>. <COUNTER> is either a single 0 bit which means one repetition, or four bits 1 D3 D2 D1, where the number of repetitions N = 2 + D3*4 + D2*2 + D1 (values 2 to 9). <CODE> has five different values: 00 - an empty space, 01 - a wall, 10 - a barrel, 101 - a place for the barrel, 111 - a barrel already in place.
• Player’s start position X (1 byte)
• Player’s start position Y (1 byte)

And so for all 60 maps.

In the file pushermaps.c there is an implementation of the simple decompressor.

When disassembling the maps also had been presented in the convenient plain text form but still compressed.

level_01        db 16h, 0Bh, 0A2h, 0DFh, 38h, 32h, 1Fh, 38h, 2Ah, 3, 0E6h
                db 12h, 0C0h, 0A5h, 0F2h, 83h, 2, 81h, 3, 0E4h, 12h, 82h
                db 25h, 6, 0CDh, 64h, 22h, 51h, 0ACh, 11h, 0A1h, 0Ah, 5
                db 0E5h, 11h, 0B1h, 14h, 82h, 29h, 82h, 31h, 0A0h, 0E1h
                db 2Ch, 18h, 0D1h, 0CFh, 80h, 0Ch, 8
level_02        db 0Eh, 0Ah, 0F6h, 58h, 0Ch, 68h, 0Dh, 94h, 0C6h, 80h
                db 85h, 2, 82h, 18h, 0D0h, 15h, 4Ch, 10h, 0C6h, 0C2h, 18h
                db 21h, 8Dh, 1, 6, 4, 39h, 10h, 0A0h, 81h, 80h, 85h, 2
                db 8, 20h, 60h, 34h, 1Bh, 0Ch, 1Eh, 0CAh, 7, 4
level_03        db 11h, 0Ah, 0E3h, 9Fh, 0Eh, 7, 0C2h, 11h, 42h, 1Fh, 8
                db 50h, 23h, 0E0h, 85h, 4, 0Ch, 1Eh, 84h, 8, 0A6h, 0B4h
                db 10h, 85h, 2, 82h, 59h, 0D4h, 28h, 14h, 90h, 0D6h, 83h
                db 0DFh, 7Ch, 0Eh, 1

… etc.

So if you’d like to implement a quick and compact Sokoban for some reason, there is a bunch of already created and nicely compressed maps.

I know that there are tons of levels for Sokoban in the internet, plus automated Sokoban solvers and similar stuff. But it still doesn’t make all that fun of dissecting the twenty years old or more binary less interesting.

The project is available at GitHub’e – https://github.com/begoon/sokoban-maps.

(photo from Wikipedia)

Now it is genuine!

Open it up! It didn’t risk to detach the keyboard, so I had to photograph from two angles.

The label “1982 ISSUE TWO” is visible at the bottom left.

There are a few lovely “extra” parts stuck on some chips. Maybe it was a result of the “last minute tuning” or maybe the previous owner did it.

The seller also gave a bunch of tapes. I haven’t seen 15 minutes cassettes before.

Extras.

So lovely books.

People had guts to put proper illustrations in manuals at that time.

I also bought a very old TV at car boot sale for a pound.

Unfortunately, this machine didn’t boot up. It beeped and printed rubbish on the screen. The seller argued for a while saying something about video synchronization problems but after the video (below) agreed to refund. Reluctantly I had to send it all back.

Programs should always have the form of paragraphs of comments that describe the intention of the program followed by paragraphs of code that implement that intention. All of the formatting should be designed to make readers as able as possible to read the code easily; the compiler doesn’t care. In particular, follow conventions of mathematics and your native language, not those you found in some random language manual. Write the comments first and then write the code, not the other way around. If you don’t know what you want to achieve and why, any code you write is, by definition, incorrect.

“Etudes” is very different from other computer science and programming textbooks. “Etudes” covers most of the important areas in a quite an unusual way. How did you come up with the idea to collect realistic problems from various areas and put them in a play situation? Could you tell us a little story about how and why the “Etudes” were born?

From 1974 to 1979, I was teaching at UC Davis. There was a new Computer Science graduate group there formed to give degrees in Computer Science, but it was really just a loose collection of faculty members from several departments. The department of Applied Science hired me as the first professor with a computer science degree to pull the program together. Along with a small group of other professors and adjunct instructors, we designed a new curriculum, put degree programs in place, and began teaching a number of new or redesigned courses.

During this redesign, I got the other faculty to agree that students needed to know how to work in the real world as well as to learn the theory surrounding computer science. By “work” in the real world, I don’t mean that I wanted to teach them “job relevant” topics; in my experience, such knowledge becomes obsolete quickly. Rather, I wanted to have them work on problems that included making imprecise specifications precise, on problems whose requirements were unclear, on problems that had real world application, on problems that required working with other people. For example, in the compiler construction course, students were required to work in two or three person teams and the same grade was assigned to everybody on a team. Part of the assignment was to write up how they divided their work and whether they thought the division was successful.

After a bit, we decided to teach a programming projects course to consolidate student knowledge of how to build complete applications. We thought of a few projects and I wrote them up. But when I looked around for more, I could only find one book that had only “toy” exercises in it – for example, print a table of all the possible binary Boolean operators. I also found one or two problems that Carnegie-Mellon used in one of its courses. So before long, I was writing “Etudes”. As it turned out, I left UC Davis soon after the book was published, so I don’t really know if they continued to use it in the projects course.

How long did it take to write the book?

“Etudes” took about 18-24 months start to finish. I had written up a few chapters for course work already. Contract negotiation was very fast, as I remember; perhaps only a month or so. I spent about a year writing the remaining chapters. Of course, I had to fit that around my other work. Then there was perhaps six months of review and revision. Once everything was in place, I only had to wait a month or two after final galley proofs went in to see my first copy.

Why didn’t you publish any other books after “Etudes”? Why wasn’t “Etudes” republished?

I left UC Davis to go work at Bell Labs. While I was there, I ran a project that built one of the first Ada compilers. We published technical papers, but there wasn’t a lot of time to write another book. After Bell Labs, I came out to Silicon Valley for a startup and, again, there wasn’t a lot of time for book writing. I also started a family and taking care of kids is (as most parents know) fairly time-consuming. And then you know how it goes.

Back in the early 70’s, Juris Hartmanis (one of my favorite teachers) told me that if you wrote a computer science textbook, you would make enough money to buy a new car. The reason was that there were at the time enough libraries who would all buy a copy of the book that the minimum income would be enough to buy at least a Volkswagen (VW Bugs were still around then!). If you got lucky, you might be able to buy a Mercedes Benz. And, of course, if you wrote something like Samuelson’s “Economics”, you could buy a new Lamborghini every year.

But for textbooks to do well, they have to be adopted in courses. Unfortunately, only a few colleges in the US adopted “Etudes”. So there wasn’t a lot of income from it. And the publisher had no real incentive to publish a new edition. There were a lot of individual people who bought copies, but that doesn’t help for textbooks.

One odd thing did happen, though. Right after “Etudes” was published, the Soviet academic publishing office reviewed it and decided to buy the copyright so that it could publish “Etudes” in Russia and the other Soviet Union states. This was a real diplomatic breakthrough, actually; quite commonly, the Soviet Union had simply ignored copyrights in the past. The payment from the Soviet purchase was a pleasant surprise.

Unfortunately, though, not a very profitable one. They paid $1,200 for the rights and that wasn’t a lot of money even in the late 70’s. I got half or $600. The good part was that “Etudes” was translated into Russian and used widely in the Soviet computer science curriculum. I work, for example, with two Russians who used it as undergraduates. Alexander, you are a third! I think that there were at least 60,000 Russian copies published.

And what kind of car did I buy? Actually, I bought a brand new piano that’s still sitting in my living room.

How did you find topics for the etudes?

There were several ways. One or two were problems I had in practical life. For example, the gasoline mileage etude and the investment etude came from things I worked on for myself. Some came from my interest in playing games with computers and ideas in artificial intelligence; obviously, the kalah and the management game were of that sort. The language problems at the end were revisions of the materials from our compiler course. Some of the problems are straight computer science just dressed up: the Turing machine, the map coloring, the format scanner.

Chapter 4 on automatic text formatting was inspired by a colleague at Lawrence Livermore Laboratory. He had written one of the very first automatic formatters and I had used it when writing my dissertation and the drafts for “Etudes”. Remember, this was before Unix was well-known and even “nroff” was pretty fancy stuff. So a problem that introduced the topic seemed reasonable.

The problem solutions were just those that “fit” within the production confines of the book. And I wanted students to see what “real” code looks like. Notice that the last problem even says that the solution is incomplete.

For each etude in the book you recommend how long it should take. Since 1978 programming languages have been “slightly” changed and become “a little bit” more powerful. But even nowadays, your timings recommendations look quite aggressive. Did you “play” the etudes yourself working over the book? Do you have your favorite one?

In retrospect, this was a big flaw. The durations are aggressive even now, even with very good languages like Ruby (my current favorite) that would do a lot of the work for you. Remember that when “Etudes” was written, Unix was just becoming broadly available, Basic, Fortran, and Pascal were probably the most common teaching languages, and C was only just becoming known. There were essentially no terminals to program on (although my students actually did mostly have terminals because of their associations with Lawrence Livermore Laboratory) so that students still prepared program card decks.

I’m not sure I have a favorite etude. I like the TRAC interpreter. It teaches a lot about the fundamental topics of programming languages, topics that are often ignored in course work. As “Etudes” mentions, TRAC and Strachey’s GPM are languages that were invented simultaneously and independently; the two author’s published papers appeared within a few months of one another. The two languages are very similar and both show how a focus on text manipulation can solve large problems. It is particularly rewarding to study them together and to understand how they agree and differ. I also like the EC loader etude because I think it solves problems in programming language translation that are still common; why techniques like this are not still in use is beyond me.

I did not actually get a chance to do a lot of problems when I was writing the book. However, when I went to work at Bell Labs, I had to teach myself Unix and all its tools. So I decided to try programming the gas mileage etude in “awk”. Oddly enough, I discovered a bug in “awk” itself that I reported to Brian Kernighan. He commented that he knew there was a problem, but nobody else had been able to pin it down. The bug turned out to be a computer science classic: “awk” was using some allocated memory after it had already been freed.

I cannot skip a question about the etude about the Vigenere cipher. There were a couple of setups in that chapter, like a “tiny” typo in the form of a missing line in the crypto text, and also a slightly wrong hint about how to crack the cipher. Of course, it did not stop people from cracking it at all, and it turned out that those “accidental complications” made this etude even more attractive. Could you shed some light on what happened there?

This is embarrassing. As it turns out, the publisher reset all the copy text before the book was printed, but I supplied the illustrations directly. Unfortunately, I supplied a bad illustration for the actual code text to crack. It wasn’t until the Russian translation that anybody noticed (or, at least, told me about it). Any lack of clarity in the text itself was just because of my writing. If that is the only problem in the text of the book itself, I would be very happy.

I presume that the preparation and maybe solving the problems in “Etudes” was not a straightforward process. Were there other “gotchas” or Easter Eggs in “Etudes”?

There are no other intentional “gotchas”. When I tried the mileage problem, I realized that it might not be as well specified as it might be. There are probably some other places that could be clarified.

Actually, writing was not too bad. Of course, making the solved problems be suitable took a lot more work than a student might put in; I didn’t want to look silly to the readers.

Fortunately, I had a lot of help and that made a big difference. The text and the illustrations were originally produced using innovative formatting and drawing programs developed by Hank Moll and John Beatty. This made the work much easier. I wrote the book itself on “coding sheets”, paper specially marked to make it easy to keypunch cards. The sheets then went to our keypunch team who actually did a lot of proofreading as they went along. When they thought I had made a mistake (or had made an outrageous pun), they would punch both an original and a correction card so that I could choose which was correct.

My publisher asked several outside reviewers to read and comment on the book drafts. One reviewer of the complete book draft made comments on almost every line. He made the book much better than it would otherwise have been. At the time, I didn’t know who this reviewer was. Later, I found out it was Steve Muchnick (well known for his compiler optimization book). I cannot thank him enough.

If you write “Etudes” today, which topics would you add?

Digital signatures and public key cryptography protocols would probably be a new topic. I would probably tune up the multiple precision arithmetic chapter. I would almost certainly change the compiler/language chapters to include more modern ideas. One thought (due originally to John Fletcher) would be to have a pair of etudes to write a Lisp interpreter in Ruby (maybe) and a simple Ruby system in Lisp. Or something else of the sort.

I would probably drop one or two of the simpler etudes. There is a lot of new artificial intelligence problems (particularly in genetic programming) that would be suitable. There are new ideas in data structures that would be fun to explore (how much better are red-black trees than simple-minded binary trees in terms of performance?). Honestly, though, I can’t tell what would make a good problem until I work on it for a while.

Most of the etudes were about implementation rather than figuring out a way to attack a problem in the first place, because you almost always gave quite solid background and enough details of a solution. Do you believe that it is also important to give students problems without even a clue of the solution? How such problems may look like? And how would you measure a degree of success of the solution?

In the real world, you are always faced with some problem to solve and the first issue is figure out what the problem really is. It is good to train students to do that analysis – and in the real world, that skill is rare – but that wasn’t really the idea in “Etudes”. That would be a course in itself. The point of “Etudes” was to give students a reasonable problem setup and to have them practice full application development. There is still a lot of room for student choices. For example, the solved problems make points about the quality of the code, the presentation of results to humans, and the performance of the final application.

There is a probably a course with problems like you propose, but then the course lectures would have to teach analysis techniques and the book itself would have to include materials on analysis. That’s a different book.

Charles, regarding your interests and work apart from “Etudes”, I know your background is compilers. For instance, you developed a compiler for the XPL language. What do you currently work on at Oracle? Still hands on programming?

I am still a programmer. One of the things I like about computer science is that you can actually build the things you think about.

My basic background has always been the theory, design, and implementation of programming languages. When I was an undergraduate, computer science didn’t exist yet so my undergrad degree is in applied mathematics. But my undergraduate thesis was on adding new features to TRAC (does that remind you of one of the “Etudes”?). I have been responsible for XPL, for a complete Ada compiler system, for the LR parser generator (with Al Shannon), for a proposal of array operations in Fortran, for a new dataflow language (with Jim McGraw), and (more recently) for the new code generator and optimizer in the Oracle programming language PL/SQL. Lately I have been working on some special projects at Oracle in the area of programming languages. If they work out, you’ll see the papers on them.

Looking at the topics in your book, it is not easy to guess which area of computer science is your favorite one. Even in the section about compilers you offered an interactive functional TRAC along with a regular procedural language. Do you have favorite areas of computer science?

My favorite computer science topics are:

1. Programming languages, particularly the specification of their semantics.

2. Data structures and algorithms and their performance.

3. Games and artificial intelligence.

In practice, my professional career has mostly been spent on the first topic. I have spent a lot of time working on compiler/language/application performance. This is the second topic, of course. When I was in middle school, I was already interested in computers and I already liked games. That’s where the third interest comes from. For me, computers are the world’s greatest toys and always have been. But I won’t see all the advances that my younger son with his iPad now takes for granted.

You can see the outgrowth of my interest in computer games in a couple of papers I wrote (with some friends) about Kriegspiel, a variant of chess. The papers are published in the “Computer Journal”. One of these is a complete program to check the legality of moves in chess and Kriegspiel. At the time the paper was written, there were a few example programs for students, but these were not very careful and missed some of the more subtle rules of chess. There were, of course, chess programs available, but this was long before open source software and at a time when a few chess programmers were trying to make a living selling chess products. So they obviously didn’t publish their secrets. So far as I know, this was the first published paper to provide a complete rules checker and I haven’t seen one since; it may still be the only refereed publication of a rules checker. You can find an audio interview about the project at the Computer Museum here in Silicon Valley.

My approach to computer science is generally to see if I can find some good theoretical model for a problem and then understand how to apply that in practice.

Over the years, have you met that programming language which you liked most for some reason?

Right now, I like Ruby a lot. I have written great gobs of C, but I don’t like its lack of security. Back in the day, I liked XPL which seemed like a reasonable version of C (of course, it was developed independently) with much more safety.

When I was a grad student, I wrote one long Snobol program. I used it to model some research that had been done in code optimization. It was possible to turn constructive proofs of theorems directly into Snobol routines. That was eye-opening. It has made me partial to languages that have interesting inference mechanisms.

I’d like to have a chance to learn and use Haskell or one of its friends for a good project. I think it would teach me a great deal. I should note that writing a toy program in a language is not the same as knowing the language. I have been doing Ruby for a year now on one project and I still learn something new every day.

When languages like C were invented, people wanted to be close to the metal. Understanding how the metal worked was the must for any professional programmer, and as a consequence, it dictated a manner of teaching programming. How would you teach programming nowadays? Not computer science in general but hands on programming?

I never understood the focus on hardware although I started doing assembly language on a machine that didn’t even have mnemonic op codes in the first assembler I used.

Actually, I think you asked the wrong question. The problem is not “programming”, but problem analysis and application development. You can program anything in any language; the question is whether the language is a good “human” fit to the problem and whether or not it is efficient enough.

Thus, I would teach principles of programming languages and then ask students to try and understand those principles in each of a number of languages. For example, did you know that Fortran always includes an interpreter? (For format statements, of course!) And what’s more, in Fortran 77, you need to implement call-by-name as part of that interpreter. You also probably want to use exceptions to build your interpreter. So the basic concepts pop up in surprising places.

(By the way, that tells you why the formatting etude is in the book. Over the years, I built two separate Fortran format interpreters.)

We started this interview from the book. Did you have your own “version” of “Etudes”? Any books and people significantly influenced on you as a programmer and scientist?

When I was in high school, a friend of my parents helped me get a summer job. That job turned into work at the local school system’s data processing center. In those days, data processing meant cards, card sorters, card punchs, and the like. But the next summer, the school system bought an IBM 1401. Because nobody else knew more than me about it, my supervisor let me be one of the programmers. Within a few weeks, I was writing payroll programs for a large public system.

This experience helped me get a part-time job in college also working with 1401’s. From then on, my life was working on computers and I decided to go to grad school in computer science instead of getting a full-time job. With more good luck than prior knowledge, I went to Cornell. The faculty there was incredibly helpful. Probably the two faculty members who taught me the most were Juris Hartmanis and John Hopcroft. I still think that Hopcroft’s language theory book is a classic (in the first or second editions). But everybody on the faculty there was extremely concerned that their students did well. And, if you look at the Cornell graduates from the late 60’s through the mid-70’s, you’ll see a really wide range of talent went through the program.

So far as books go, my favorites are probably those by Knuth. I really recommend reading the “Art of Computer Programming”. You won’t understand most of it the first time. My theory is that you read it over and over, a few pages at a time. Each time you look at it, a little more sinks in.

I also read all of Newman’s “The World of Mathematics” which I first saw in the 10th grade. This gave me a lot of historical background in math and mentioned enough about computers that I was helped along. I think this is still a useful introduction for young people (not to mention old people!).

Everybody comes to the world of computers differently. Maybe nowadays it is much easier because computers are everywhere. How did it happen that you chose computers as your profession in the days of mainframes and punch cards?

Well, some of the answer is in the previous question. My father owned a small hand-crank desk calculator with a design similar to that described by Felix Klein in his mathematical education book. My father’s came from the Orient (he got it during WWII, I assume), had no instructions, and was covered in Japanese symbols – but fortunately Arabic numerals. I spent many hours figuring it out.

When I was in middle school, I already liked computers and mathematics. Sometime along in there I got a toy “computer” that was made out masonite, old-fashion clasp paper clips, screws, light bulbs, batteries, and wires. It was really what would be called a gate simulator today, but you could build some very simple circuits with switched inputs and lit-up light bulbs as outputs. I was fascinated. I also found some other books about early computers and they were very interesting to me.

In my first year of high school, I went to NSF summer camp where we took some college course in algebra and group theory, learned (well, in my case, was mystified by) some finite differences, and got to program a computer. I also learned to play bridge because two of girls wanted to have a fourth (when you are in high school, girls are a powerful incentive!).

So by the time I got out of college, between the experiences I have described and some college courses (notably with Tom Cheatham), I was pretty much hooked.

Wrapping up, there is a saying about the three things a man must do. Can you name your three things a programmer must (or should) implement?

I’m not sure I have three things that I think programmers should implement. But I know some things that I think every programmer should either know or be able to do.

1. Some ability to do formal mathematics. The level necessary to understand Hopcroft and Ullman combined with a little graph theory is enough. Discrete mathematics is essential; calculus is only necessary for folks working in special application areas.

2. Some ability to write clearly in their native language. Dijkstra said that if a person couldn’t write their own language, they couldn’t write a correct program. (Well, that’s what I hope Dijkstra said!). Writing programs is essentially the same as writing a non-fiction essay. If you can’t make the step-by-step connections clearly in your own language, why would we think you could in C, for example?

3. Remember that a program is primarily for communication with humans, not computers. When you write a program, the computer will do whatever you say. You must convince the humans who read the program that what you have asked the computer to do is the correct thing to do. Remember, the computer doesn’t care about correctness.

4. (Bonus answer – 4 for the price of 3). The answer to the last question means that programs should always have the form of paragraphs of comments that describe the intention of the program followed by paragraphs of code that implement that intention. All of the formatting should be designed to make readers as able as possible to read the code easily; the compiler doesn’t care. In particular, follow conventions of mathematics and your native language, not those you found in some random language manual. Write the comments first and then write the code, not the other way around. If you don’t know what you want to achieve and why, any code you write is, by definition, incorrect.

If you look at the last “Etudes” chapters, I hope that you can see that even back then, I was trying to apply these principles.

Thanks, Charles for the interview answers. We (fans of “Etudes”) still hope for your new books or other publications and wish you all the best. In meanwhile, if you, my dear readers, have not seen “Etudes” so far, please grab a copy and take a look. It is worth it.

■

// Charles Wetherell, Alexander Demin

// August 2012

P.S. If you have questions to Charles, please, put them in comments below.

For instance, to form a stub number for a 16- or 32-bit value, people usually use words from the following set:

ABBE
ACED
BABE
BADE
BEAD
BEEF
BODE
CADE
CAFE
CEDE
COCA
CODA
CODE
DACE
DADO
DEAD
DEAF
DECO
DEED
DODO
FACE
FADE
FEED
FOOD
OBOE

For example:

DEADFACE
ACEDFEED  

If we don’t care about the symmetrical split, the vocabulary is a bit wider:

ABBE
ABE
ABODE
ACCEDE
ACCEDED
ACE
ACED
ADD
ADDED
ADO
ADOBE
BAA
BABE
BAD
BADE
BAOBAB
BE
BEAD
BEADED
BED
BEDDED
BEE
BEEF
BOA
BOB
BOBBED
BODE
BODED
BOO
BOOBOO
BOOED
CAB
CACAO
CAD
CADE
CAFE
CEDE
CEDED
COB
COCA
COCOA
COD
CODA
CODE
CODED
COFFEE
COO
COOED
DAB
DABBED
DACE
DAD
DADO
DEAD
DEAF
DEB
DECADE
DECAF
DECO
DECODE
DECODED
DEE
DEED
DEFACE
DEFACED
DO
DOC
DODO
DOE
DOFFED
EBB
EBBED
EFFACE
EFFACED
FAB
FACADE
FACE
FACED
FAD
FADE
FADED
FED
FEE
FEED
FOB
FOBBED
FOE
FOOD
OAF
OBOE
ODD
ODE
OF
OFF

For example:

BADCOCOA
BADCODED
FADEDDOC
CODEDBOB

On 64-bit systems we can create almost complete sentences.

EBBEDDEADBAOBAB

Does anybody know what to say to the next, 128-bit generation?

In short – it is awesome! I’m not talking about the material they provide. It is great without any concerns. I’m talking about the technology of online education.

Briefly. The course lasts six weeks. Each week we are given video lectures, a hour and a half in total, broken down into 10-20 minutes episodes. The videos are interactive, and sometimes you’ll get a quiz along with watching. For each episode there are two PDFs. The first has live commentaries written by hand. They appear on videos along with lector’s voice. The second is a regular PDF with slides. You can watch videos as many times you want: in the browser or downloading it. English subtitle are available. There is a cool feature in the online video viewer – you can change the speed from 0.5 to 1.5, so it turns to too much “blah-blah”, there is a way to speed it up.

Also you’re given with a list theoretical questions and one programming question each week. You only have a few attempts for submission. The best answers count.

There is no real deadline but submissions after the “official” deadline will be graded with 50% “discount”. So, if you’re up the certificate after the course, it’s worth doing it in time.

At the very end there is a final exam. You’ll have 3 hours to complete.

There are forums available where you can communicate with other students and course owners.

Subjectively, I’d make the course a bit “tougher”, but in general it has perfect balance for a couple of evenings a week.

Conslusion: Strongly recommended. Not this particular course because everybody has different interests, but the technology in general.

Nowaday if someone says that he or she has no money for an expensive and famous university, you can recommend to save little money for internet and an inexpensive computer. That’s it.

I’m up to try 6.002x: Circuits and Electronics next. Just curious how they do a practical part for electronics online.

From 64MB to 32GB.

An idea occured to join them all together into RAID, for example, a concatenated disk. In OSX it can be done in a few clicks. The idea is pretty silly but it was interesting to try.

I’ve bought an extender with 12 ports.

Satechi 12 Port USB Hub with Power Adapter & 2 Control Switches

Plugging them all in…

Start the Disk Utility and check that all the sticks are recognized.

Create a “concatenated disk” named “Crazy RAID” and add the sticks into there.

Confirming…

Wait for five minutes and it’s done.

“df” perfectly sees the drive.

Now we can copy something there and check the transfer rate.

Of course, this is just a toy without real usage. All the sticks have different capacity, so the concatenated disk is the only choice in terms of the RAID features. Write speed depends on a particular stick which is currently in use. On concurrent read we may gain some speedup.

But it looks cool!

Below there is a bunch of photos of things which had caught my eye. Of course, there is much more stuff in the museum.

The real BBS, even if running Windows.

The ancestors of an era.

The NeXT CUBE.

Brothers! This is an 1200 bps modem, or 300 full duplex.

Have you seen The Matrix? Remember the modem they used for teleportation?

I’d be surprised if there isn’t Raspberry Pi in the British museum, so here it is.

A portable laser-disc storage.

Oh, I should probably buy this one and relax.

Brothers! Fasten your seat belts. This is the whole room of working BBC Micros. Books and manuals are ready to help. You can sit and program.

And I did.

BBC Micro inside.

Books are everywhere, new ones, old ones.

RML 380Z

PET Commodore

ELIZA can ask questions and answer.

The history of computer memory, from bulbs to SIMMs, DIMMs and all the rest of it.

A flash drive from the past.

Computer trainers. I love these charmy devices.

Would you like to study on this one? I would.

Ergonomic keyboards weren’t invented yesterday.

A very expensive computer for business from the 80-th.

There are a few lovely cassettes on a huge bookshelf with manuals. It turns out that they were used not only at home.

Go to a middle heavy weight. Now floor-standing machines instead of desktops.

Cray

PDP-8

PDP-11

A XEROX workstation.

VAX. I “saw” this guy only via a terminal when I was studying.

A graphical station.

Terminals, terminals.

A hall of epoch-making computers.

Recognize this guy?

In the corner? We (Russians) knew it as a Zonov’s clone implemented on Soviet parts, but this is the original.

In action.

It turns out that there were hell a lot of machines in the keyboard case form factor.

This computer had launched Microsoft.

This one had started Apple.

You can land your bottom on the bench and play rare games of epoch-making rare computers form the 80-th.

A room to play the Flight Simulator.

The mainframes.

UNIX. The Beginning.

A real radar monitor.

By the way, a true computer museum cannot be without WiFi, so here it is.

PDP-11

An analog computer. In digital computers we break down tasks into primitive binary operations. In analog computers we have to use operational amplifiers to add, subtract, integrate or differentiate electrical signals.

Convenient, however.

A new, not yet complete exposition.

But something is already there.

Punch card machines.

A (attention) calculator!

The archives of Computer Weekly.

Brothers! This is just awesome, the must see place for computer enthusiasts. Unfortunately, I had only forty minutes for everything, so maybe I’ll go there once again.

This device allows to connect legacy RS-232 devices to the world of TCP/IP. We used such things years ago developing software for POSes and ATMs. Just change a dialing script from something like ATDT12345678 to ATD192.168.1.1 and off you go.

Have no idea what to do with this now. I even have no proper RS-232 port, only via a USB cable. Interestingly, back those days such devices were very expensive.

Programming 32-bit Microcontrollers in C: Exploring the PIC32 (Embedded Technology)

I stopped programming for PICs about ten year ago when we were developing a GSM phone based alarm system. We used the PIC16 chip and a DTFM decoder (be the way, the feature of controlling the alarm system dialing to it and using DTFM signals was pretty awesome that time) and a bunch of sensors.

I also worked with PIC12. I was stunned of just an 8-pin chip programmable in C!

Then I moved to the UK and left a lot of stuff at home. But recently it all has come back when I built Maximite and also bought Raspberry Pi. Since then the technology has stepped forward, of course. Now PIC32 chips are based on the MIPS kernel, almost fully 32-bit and have a primitive virtual memory capabilities. It allows creating simple operating systems when the kernel has its own protected memory from users processes. For example, RetroBSD perfectly works on Maximite.

Back to the book. I wanted to refresh my PIC knowledge and opened the book. To my surprise I swallowed it in a couple of evenings non-stop.

The book is organized perfectly and suitable for people having minimal experience in microcontrollers. But it covers serious topics such that:

• memory model
• interrupts (for example, timers or I/O)
• serial or parallel interfaces (for example, connecting to a LCD display or a PS/2 keyboard)
• ADC/DAC (for example, a temperature sensor or an analog joystick)
• generating black/white video signal (for example, composite or VGA) using non-trivial DMA-based data transfer for a memory buffer to the SPI channel
• SD/MMC cards interface
• implementation of the FAT
• sound generation via PWM

All example are complete and working. It can be even better if you have a companion CD of the book.

The only little issue was that the author used MPLAB and the C32 compiler (remember, the book was released in 2008), but Microchip strongly recommend migrating to MPLAB-X (which, by the way, the only choice for Linux and OSX) and the XC32 compiler based on GCC. Of course, the project files can be easily converted to the new MPLAB-X format, the book lacks examples of simple Makefiles. But it compensates it with exhaustive screenshots and commentaries about configuration windows, the debugger and the simulator. It is possible to read the book even without a computer.

If you read the book, you’ll understand what can be implemented with PIC32 and how things work in it. If you decide to implement a project, you have enough examples plus understanding how to start the crystal.

The conclusion: Highly recommended.

When I found that the PIC32 chip could be made to use a VGA display and keyboard, it was only natural that I should try and recreate one of the earlier computers that I had so much fun with in my youth.

Geoff built a microcomputer, Maximite, using a PIC32 chip. Maximite is a complete appliance utilizing modern interfaces. It uses VGA for video, PS/2 for the keyboard, SD cards as storage, USB for communicating, and it runs BASIC! Doesn’t it remind you something? In my view, Maximite is a perfect tool to study microelectronics allowing to touch how hardware meets with software.

Today we can ask Geoff a few questions in person. Geoff lives in the suburb of Kensington in Perth, Western Australia. Perth is a long way from anywhere but with the wonders of the Internet and FedEx the world is a much smaller place now, so it doesn’t stop Geoff to design and produce great hardware and software.

Hi Geoff, thanks for the interview. Let’s begin with a light question – why “Maximite”? How did you invent this name?

At the beginning of the project I wanted to call it The Mighty Mite meaning something that was small (mite) that was also powerful (mighty). When the design came to be published in Silicon Chip magazine the editor discovered that the name had been trademarked by a food manufacturer, so he suggested Maximite as an alternative. By then I could think of nothing better, so that became the name. As it turned out, that is also the name of an explosive which, in a strange sort of way, is also appropriate.

Maximite is very popular. Do you know at least rough figures of how many of them were built? Also when and how did you realise its incredible popularity?

My standard answer is that “thousands are in use” but I believe that a more accurate number is somewhere between 3000 and 4000. About half of them have been built by people with a soldering iron and the other half purchased as an assembled board from companies who make copies of the design.

I realised that it would be popular when I looked at the number of visitors to my website, in the first few days after the magazine article appeared the number of unique visitors to my website jumped by 100 times before settling down to a long term increase of 20 times the pre Maximite days. What was interesting was that my URL did not appear in the first magazine article so many people read the article and immediately used Google to find out more.

When I started on the design of the Maximite I was thinking of the computers of the 80’s which provided me with a lot of fun with back then. They were very popular in their day and, as the modern computers have become more complicated and difficult to use, there was a vacuum left behind. I believe that the key to the success of the Maximite was that it filled that space.

For Maximite you had to build hardware and software. This is quite an unusual position nowadays for people working with computers. Professionals in software rarely look at hardware and vice versa. Where are you here? Are you more a hardware engineer by nature or a programmer?

Yes, that is one of the things that I have found since the Maximite design was published. People tend to look at it through the eyes of a hardware engineer and do not appreciate the software - they want to add electronic features without regard to the firmware or how the BASIC language could accommodate them. There is always the reverse too, people who only think in terms of the software. I think that I am best described as a programmer but with a strong electronic background.

Another item that is often missed when people try to change the Maximite is its “look and feel” (ie, how the user uses it). This must be consistent with the hardware and software. All three are part of the integrated whole and this is often not appreciated by either the hardware or software specialists.

I personally was amazed of how solid and complete the Maximite project is. You designed the board, the case, software, and the kits of Maximite are available online. In the world of individual homebrew projects people usually don’t tend to finish off their projects fully to such a polished shape because they loose interest. How did you manage that?

This primarily came from my desire to have the project published in Silicon Chip magazine. Most of their projects are a complete “product” designed by a professional engineer and I had to meet that standard if I wanted them to publish the Maximite project. This attention to detail is unusual today as magazines tend to only half finish the project, things like documentation, a box to put it in, a nice front panel and so on are often ignored.

I find that the discipline of designing projects for the magazine is quite good. I have a number of personal projects (ie, ones not published by the magazine) that are still sitting around today in a half completed form because I do not have that discipline operating on me.

Geoff, you started your microelectronics hobby when you retired. How much time a day you usually spend on it? Again, you delivered the Maximite project in a such complete form, so seems it required some time to put everything in place.

Yes, I have spent far too much time on it since I have retired, but interestingly it was something that I had to do. It is as if there was a design engineer hiding inside of me that could not get out while I was following my professional career (which was not in electronics).
The moment I retired he broke forth.

With the Maximite things seem to come in bursts, I would spend twelve hours a day for a month or two then have a break for a few weeks. It is good that I enjoy working on projects like this because they do take up a lot of time. I estimate that the Maximite has taken about nine months of solid eight hours a day, forty hours a week to get it to where it is now.

MMBasic. I know you struggled to find software for Maximite. A few existing implementations of BASIC you tried didn’t really fit, and you ended up with your own implementation of BASIC. Could clarify the licensing and distribution model of MMBasic? Any plans, maybe, to develop a Maximite framework or a library allowing to deal with Maximite peripherals in the form of device drivers? It could be useful in developing alternative firmware for Maximite, for example, running Lua or RetroBSD.

Yes, when I started on the Maximite project my hope was to find a free version of BASIC in the Internet and use that. But I found that the good implementations of the language were not available freely or had some other issue. So I decided to write my own.

Initially I released my version of BASIC (called MMBasic) under the GPL GNU licence but I had a bad experience with one commercial organisation who removed my name from the software and claimed it as theirs, they even changed the copyright to their name. So I created my own license where the software is freely available but it cannot be redistributed without my agreement.

This has worked out very well, a number of universities and commercial organisations are using it and I have an agreement with them covering what they will do with it. This is much better than allowing someone to anonymously download my software and then see it appear on the Internet with their name and copyright.

At this time I do not have plans to develop a loadable device driver type of architecture. The main reason is that the PIC32 chip does not have enough memory to do all the things that it already does (VGA, keyboard, USB, etc) and implement a loadable device driver system.
There are more powerful chips around but they are impossible to hand solder so I have stayed with the PIC32. Products like the Raspberry Pi can have loadable device drivers but they also have a complex operating system - just what I was trying to avoid with the Maximite.

Let’s move a bit to your personal background. When and how did you study to solder, for instance? I know many people looking for some experience in electronics but they are afraid of soldering seeing it potentially difficult or even impossible. How would you recommend people starting messing around with electronics hands-on?

I cannot remember when I first started soldering but it was probably around age 12. I am probably the wrong person to ask but I cannot see the problem with soldering – you only need a cheap soldering iron and some wire solder.

One of the great things with electronics is that it can be a very cheap hobby. I would recommend starting with a simple and cheap project like flashing a light or adding a clock chip to the Arduino and just buy the bits including the soldering iron. You might make a mess of the first project but that is not important and by the time you have reached the third project you will regard yourself as almost professional.

Would you recommend any books or websites where beginners could make their first easy steps into the wonderful world in electronics? What kind of minimal level of knowledge could be enough to design and build, for example, Maximite?

My recommendation would be to start with a magazine subscription. There is Silicon Chip (in Australia), Elektor (in Europe) and Nuts and Volts (in the USA). All of these will accept international subscriptions. If you are starting with no knowledge at all probably Nuts and Volts would be the best. When you subscribe to a magazine you will have a constant stream of ideas and projects to tempt you into trying something different and fun. So a subscription is well worth the cost.

The level of knowledge required to build the Maximite is not great because it comes already designed and with working software. On the other hand, to design it you would need a great amount of expertise. I have a lot of expensive test gear, I trained as an electronics engineer in my early days and I have about 30 years of experience in systems programming - all that is necessary to create something like the Maximite.

Now we have plenty websites providing a marketplace for individual hardware developers to sell their ideas and designs, for example, SparkFun.com. The internet allows people to get some money from hobby. You, as a person, whose hardware designs are being wildly sold over the internet, do you think it is feasible to live on income from such business? or volumes are still minor?

It would be very hard to make a living income from that sort of thing. There are a few, like the designers of the Arduino, who are probably doing OK but there are many more who have to keep another job so that they can pay the rent. Part of the problem is people like me who produce sophisticated designs and then give them away for free, it is hard to compete with that and your customers come to expect products or software as a very low price.

A bit of personal stuff. Can you name any people, for example, engineers or programmers, which you did or would like to learn from?

The programmer who was the most influential on me is Brian Kernighan who, along with Dennis Ritchie, designed the C programming language and wrote a book on it called “The C Programming Language” in 1978. Even today that book is a wonderful introduction to the C programming language and well worth a read if you want to get into programming in C. When I write my magazine articles and web pages I always remember his clear and easy writing style and try to emulate it in what I do.

Probably the most influential engineer is Jim Rowe who was creating projects and writing for electronics magazines in Australia when I was a young teenager. He is still doing it today 50 years later and I always find his projects well designed and his magazine articles clear and easy to understand. He is someone else who I try to emulate in my projects and articles.

By the way, how did you come to the world of computers and engineering? What was your first computer or programming language?

This will date me somewhat (I am 64 years old). I remember the Altair computer when it was first released and a little later the company I was working for became the distributor for Intel in Australia. I was studying as an electronics engineer at the time and my first computer was one that I hand soldered using the Intel 8080 chip and 512 bytes of RAM (yes bytes).

Back then BASIC was the only language that you could use because of the limited memory capacity and hardware. Much later, when we had disk drives, you could use more sophisticated languages and in my time I have written programs in most of them (Fortran, COBOL, PL/1, assembler, Pascal, etc).

As you can see, I was involved in electronics, small computers and BASIC a long time ago. So, when I found that the PIC32 chip could be made to use a VGA display and keyboard, it was only natural that I should try and recreate one of the earlier computers that I had so much fun with in my youth.

Wrapping up, I’d like to ask you to shed some light on your future projects? Is Maximite finished or we should wait for the Maximite with Ethernet or WiFi or something?

There is the new Colour Maximite that will be published in the September 2012 issue of Silicon Chip. It will have eight colours on a VGA display, stereo synthesised sound and an Arduino compatible connector along with all the other features of the original Maximite (USB, BASIC language, etc). That has taken quite a lot of my time and energy to create, so after it is published I might take a break for a while before looking around for something else.

There is a couple of exclusive photos of the new colour Maximite.

Thanks, Geoff, for the interview and for Maximite. Looking forward for your further projects.

I’m personally incredibly excited that today people have started to realise that teaching computers is not only about Word and Excel, but also about understanding what is inside and how all these guts actually work. And projects like Raspberry Pi and your Maximite help enormously get beginners a bit closer to the metal.

■

// Geoff Graham, Alexander Demin

// August 2012

Palm-Size Portable Mini Travel 2 in 1 Wireless N 802.11n/g WLAN Network Router / Client Adapter

This is called WA-6220.

The official website says:

WA-6220-V1 WLAN 11n Router is designed for people on the go, user can carry it travelling around and work at anywhere. It is compliant with 802.11n specifications, up to 300Mbps data rate, provides multi-functional capabilities, particularly the high performance throughput and high-quality security. Incorporating fast Ethernet ports and is compatible to other wireless (802.11a/b/g/ n) networking device, it enables your whole network sharing a high-speed cable or DSL. Its high performance is ideal for media-centric applications like streaming video, gaming and Voice over IP technology.

• Provides wireless speed up to 300Mbps data rate.
• Supports WEB based management and configuration.
• Supports UPnP, IGMP Snooping & Proxy.
• Supports Dynamic DNS, NTP client service.
• Supports Log table and remote Log service.
• Supports Setup Wizard mode.
• Supports Wireless schedule.
• Compatible with IEEE 802.11a/b/g/n Specifications.
• Supports multiple AP, provides maximum 5 groups of SSID.
• Supports IEEE 802.3x full duplex flow control on 10/100M Ethernet interface.
• Supports bridging, routing and WISP functions between wireless and wired Ethernet interfaces.
• Supports IEEE 802.1x, 64-bit and 128-bit WEP, WPA, WPA2 encryption/decryption and WPA with Radius.
• Supports DHCP server to provide clients auto IP addresses assignment and static DHCP functions.
• Supports DHCP client, static IP, PPPoE, PPTP, L2TP of WAN Interface.
• Supports firewall security with port filtering, IP filtering, MAC filtering, port forwarding, URL Filtering and DMZ hosting functions.
• Supports AP mode, Client mode, WDS, AP+WDS and Universal Repeater.
• Supports QoS which controls the bandwidth by IP or MAC address.
• Supports Denial-of-Service.
• Supports WPS (Wi-Fi Protected Setup).
• Supports WAPI (Wireless Authentication Privacy Infrastructure).
• Supports Green Ethernet, IEEE802.3az draft2.0, EEE (Energy Efficient Ethernet).

It is the matchbox in size.

And can be powered from USB (I connected it directly to the laptop).

Front.

The mode switch and reset button.

The quick user guide.

Unfortunately, I haven’t met a hotel with properly installed WiFi coverage. Many can put a cable, but proper WiFi coverage with decent speed – no one. But this device makes you an ad-hoc instant router in minutes with no pain whatsoever.

P.S. By the way, a question – is WiFi hardware (chipset) in laptops, for instance, and regular WiFi access points different in any way? If the connection is duplex, it shouldn’t be. So, how to turn a laptop into a proper WiFi access point then? I tried things like Connectify on Windows 7, but it didn’t really work properly. And what should I also do on Linux and OSX?

Initial re-partitioning

At the time of this writing the standard Debian images (stable) distributed from the official Raspberry Pi website were built for 2GB SD cards. It is possible to copy them to larger cards and it will work but space above 2GB will be wasted.

There are a few simple steps to re-partition the card after the first boot.

Images:

http://raspberrypi.org/downloads/

Step by step:

https://projects.drogon.net/raspberry-pi/initial-setup1/

In short:

printf "d\n3\nd\n2\nn\np\n2\n157696\n\nw\n" | sudo fdisk -cu /dev/mmcblk0
sudo shutdown -r now
sudo resize2fs /dev/mmcblk0p2

Troubleshooting

http://elinux.org/R-Pi_Troubleshooting

GPIO

General info

http://elinux.org/Rpi_Low-level_peripherals

WiringPi

https://projects.drogon.net/raspberry-pi/wiringpi/

• Pins
  • Special Pins
• Download and install
• Wrappers
  • Ruby

RPI.GPIO

http://pypi.python.org/pypi/RPi.GPIO

Homepage: http://raspberry-gpio-python.googlecode.com

How to install:

cd /tmp
wget http://raspberry-gpio-python.googlecode.com/files/RPi.GPIO-0.2.0.tar.gz
tar xzvf RPi.GPIO-0.2.0.tar.gz
cd tar xzvf RPi.GPIO-0.2.0
python setup.py install

Sample program:

import RPi.GPIO as GPIO
GPIO.setmode(GPIO.BCM)
pin = 4
GPIO.setup(pin, GPIO.OUT)
while True:
    GPIO.output(pin, True)
    GPIO.output(pin, False)

Benchmarking Raspberry Pi GPIO speed

http://codeandlife.com/2012/07/03/benchmarking-raspberry-pi-gpio-speed/

Example with 2 LEDs

https://projects.drogon.net/raspberry-pi/gpio-examples/tux-crossing/2-two-more-leds/

GPIO Port on the Raspberry Pi

http://chrishatton.org/archives/88

SPI/I2C 3.2 kernel

This kernel is based the 3.2 branch. It comes with I2C drivers and a bunch of staging drivers (for example, WiFi).

http://bootc.net/projects/raspberry-pi-kernel

I2C

Quick2Write

• http://quick2wire.com/
• http://quick2wire.com/blog/
• An experimental GPIO Python library for the Raspberry Pi

Linux I2C interface

http://kernel.org/doc/Documentation/i2c/dev-interface

I2C examples

I2C examples

Example I2C Code

echo ds1307 0x68 > /sys/class/i2c-adapter/i2c-0/new_device

I2C and the Raspberry Pi

Explanations why and how I2C was added to BootC RPi kernel:

http://bootc.net/archives/2012/05/19/i2c-and-the-raspberry-pi/

MCP23017 I2C IO expander

http://nathan.chantrell.net/20120519/raspberry-pi-and-the-mcp23017-i2c-io-expander/

Complete I2C setup on Raspberry Pi

http://robot-electronics.co.uk/files/rpi_i2c_setup.doc

Set up for using i2c with Raspberry Pi running the standard Debian squeeze distribution.

sudo dpkg-reconfigure tzdata
sudo apt-get update
sudo apt-get upgrade 
sudo apt-get install ca-certificates

Install hexxeh’s rpi-upgrade tool https://github.com/Hexxeh/rpi-update#readme:

sudo wget http://goo.gl/1BOfJ -O /usr/bin/rpi-update
sudo chmod +x /usr/bin/rpi-update
sudo apt-get install git-core

sudo rpi-update

Restart rpi for updates to finish:

sudo shutdown -r now

Download Chris Boots’s kernel with the i2c drivers in as a Debian package from his website http://bootc.net. Navigate to where the Debian package was saved using command line and install the Debian package.

dpkg -i linux-image-3.2.18-rpi1+_5_armel.deb

Copy /boot/vmlinuz-3.2.18rpi1+ to /boot/kernel.img:

sudo cp /boot/vmlinuz-3.2.18rpi+ /boot/kernel.img

Restart RPi again:

sudo shutdown -r now

Download i2c-tools:

sudo apt-get install i2c-tools

If you run ls /dev/i2c* you will not be able to see any i2c ports (e.g. /dev/i2c-0) listed. At start up the i2c ports will not be active. Make it active using modprobe:

sudo modprobe i2c-dev

Now look to see if your i2c ports exist in /dev:

ls /dev/i2c*
i2cdetect -l
dmesg | grep i2c

You should see two i2c ports listed named /dev/i2c-0 and /dev/i2c-1. /dev/i2c-0 is the one we will be using. Another way of listing them is to run i2cdetect -l (a tool that came with the i2c-tools you installed earlier).

Change permissions of the i2c-0 port to let you access:

sudo chmod 666 /dev/i2c-0

You should now be able to download and run some of our example c code. You will need to modprobe i2c-dev and change the permissions of the i2c port every time you boot up as these setting are not saved by default.

Checking Firmware Version

Firmware should be up-to-date. Its update is harmless and cannot brick your RPi.

/opt/vc/bin/vcgencmd version

RPI update

Method 1

https://github.com/Hexxeh/rpi-update

wget https://raw.github.com/Hexxeh/rpi-update/master/rpi-update -O /usr/bin/rpi-update && chmod +x /usr/bin/rpi-update

sudo apt-get install ca-certificates
rpi-update

Method 2

http://iroylabs.blogspot.co.uk/2012/06/raspberrypi-mouse-and-keyboard-does-not.html

wget https://raw.github.com/Hexxeh/rpi-update/master/rpi-update
chmod +x rpi-update
mv rpi-update /usr/bin/rpi-update
sudo rpi-update

Keeping Raspberry Pi fresh

http://wrightrocket.blogspot.co.uk/2012/06/keeping-your-raspberry-pi-fresh.html

USB to serial cable

Olimex USB-SERIAL-CABLE Olinuxino console cable alternative to Raspberry Pi.

• three wires for Breadboard connection
• SERIAL output 3.3V voltage levels
• GND=BLUE, RX(INPUT)=GREEN, TX(OUTPUT)=RED

BLUE (GND) -> 0v GREEN (RX) -> GPIO 14/TxD (BCM), or 15 (wiringPi) RED (TX) -> GPIO 15/RxD (BCD), or 16 (wiringPi)

Driver for Mac OSX Lion

http://xbsd.nl/2011/07/pl2303-serial-usb-on-osx-lion.html

cd /tmp
wget http://xbsd.nl/~martijn/log/osx-pl2303.kext.tgz
tar xvzf osx-pl2303.kext.tgz
cd osx-pl2303.kext
sudo cp -R osx-pl2303.kext /System/Library/Extensions/
cd /System/Library/Extensions
chmod -R 755 osx-pl2303.kext
chown -R root:wheel osx-pl2303.kext
cd /System/Library/Extensions

kextload ./osx-pl2303.kext
kextcache -system-cache

Check:

kextstat -b nl.bjaelectronics.driver.PL2303

Index Refs Address            Size       Wired      Name (Version) <Linked Against>
   74    0 0xffffff7f808ee000 0xb000     0xb000     nl.bjaelectronics.driver.PL2303 (1.0.0d1) <73 34 5 4 3>

Unload the driver (if needed)

Check “kextstat” first, and if “nl.bjaelectronics.driver.PL2303” is there, it can be unloaded by:

sudo kextunload /System/Library/Extensions/osx-pl2303.kext/

Driver as a package

http://changux.co/osx-installer-to-pl2303-serial-usb-on-osx-lio

PL-2303 cables on Mac

http://planet-rcs.de/article/mac_serial_port/

WiFi

http://raspberrywifi.com/

ASUS N10 WiFi adaptor

Kernel 3.2.21-rpi1+ already has this driver, but it requires firmware.

1. Add the following line to /etc/apt/sources.list:

   deb http://backports.debian.org/debian-backports squeeze-backports non-free

2. Update the package index:

   sudo apt-get update

3. Install firmware-realtek deb package:

   sudo apt-get install firmware-realtek

Check:

sudo iwlist wlan0 scan

Post-setup:

sudo vi /etc/network/interfaces

auto wlan0
iface wlan0 inet dhcp
wpa-ssid "your_ssid"
wpa-psk "your_password"

Cool gadgets

VGA over serial port

http://hobbytronics.co.uk/serial-vga

Cardboard case

http://squareitround.co.uk/Resources/Punnet_net_Mk1.pdf

Blogs about Raspberry PI

• https://projects.drogon.net/raspberry-pi/
• http://bootc.net/projects/raspberry-pi-kernel
  • http://bootc.net/
• http://raspberrypi.homelabs.org.uk/
• http://pi.gadgetoid.co.uk/
• http://codeandlife.com/topics/raspberry-pi/
• http://blog.mx17.net/category/hardware/raspberry-pi/

Useful sites

• https://projects.drogon.net/
• http://hobbytronics.co.uk/
• http://codeandlife.com/
• http://sparkfun.com/
• http://gadgetoid.com/

GMC-4 is cloned from other microcomputers, and its instruction set is compatible with:

• FX-microcomputer R-165
• Tandy Microcomputer Trainer (english)

Links to various resources about GMC-4

• Official magazine issues

  • http://otonanokagaku.net/feature/vol23/index.html
  • http://otonanokagaku.net/magazine/vol24/index.html
  • http://otonanokagaku.net/english/magazine/vol24/index.html (english)

• Review and documentation

  • Programming the Gakken GMC-4 Microcomputer, architecture and instruction set (english)
  • GMC-4 Microcomputer Manual (english)
  • About GMC-4 (introduction, instruction set, simulator, compiler)
  • GMC-4 microcomputer (english)

• Firmware

There are no sources available of the original GMC-4 firmware.

• Hardware

  • Schematic (version 1, version 2)
  • Keypad connector for the loader (upside, downside, schematic)
  • USB loader 1 (assembled, interface board, schematic, port layout, firmware sources, 74HC4051 datasheet)

• Blogs

  • http://blog.nshdot.com/search/label/gmc-4/
  • http://airvariable.asablo.jp/blog/cat/gmc-4/
  • http://100year.cocolog-nifty.com/blog/gmc4/
  • http://www.geocities.jp/yasuho68k/gmc4/ (a bunch of sample programs)
  • http://d.hatena.ne.jp/yasuho/searchdiary?word=%2A%5B4%A5%D3%A5%C3%A5%C8%A5%DE%A5%A4%A5%B3%A5%F3%5D

• Simulators

  • GMC-4 simulator (screenshot)
  • FX-microcomputer R-165 simulator (screenshot)
  • On Android

• Loaders

  • GMC-4 Microcomputer USB loader based on UBW (english)
  • GMC-4 USB loader based on the AVR controller
  • GMC-4 loader based a PIC development board

• Assemblers

  • http://www.musashinodenpa.com/misc/GMC4/
  • http://slashdot.jp/~roto/journal/481254
  • https://github.com/jasperla/gmc4-as/
  • http://fxm.piroyan.com/
  • http://d.hatena.ne.jp/nicotakuya/20090702/1246521731

• Compilers

  • C Compiler
  • BASIC Compiler

• Replications

  • avr-gmc-4 (AVR) (sources)
  • devkitARM (ARM) (screenshot, sources)

• Links collection from the Official Wiki

  • http://www15.atwiki.jp/gmc4/pages/12.html

This is a USB Bit Whacker, a development board based on PIC18. Its default firmware allows to control I/O pins via simple commands over a virtual serial port.

Schematic.

A prototype.

I was lazy to etch the board, so soldered manually. Looks unsightly but works.

In action.

After all, GMC-4 is THE thing!

P.S. The sources are available on Github – gmc4-loader. I suspect that in Python or Ruby it might be much nicer, but it was a nice exercise programming serial ports in UNIX.

Unless the multithreaded code is going to be clearly faster, then writing single-threaded code will save you a lot of headaches.

I recently purchased his book about concurrency in C++. To be honest I haven’t met better material so far about C++ 2011 memory model. Today we have an opportunity to ask Anthony a few questions, and, of course, particularly about C++.

Hi Anthony, thanks for the interview, and let me begin from afar. Programming, computers, C++… Why had you chosen this way? What dragged you on this route of bits and bytes?

I’ve always been interested in computers, and it turned out that I was reasonably good at programming them. We had a Sinclair ZX81 at home, and a BBC Micro at school when I was around 7 or 8, and it all started from there. I used to examine the source code of the games to see how they worked. When it came time to apply for jobs after my degree, I was quite clear in my mind that I wanted to do programming. It was fun, interesting, challenging, and you got paid for it!

I suspect that C++ wasn’t the first language you’ve learned. Have you ever programmed in some unusual languages at early days?

C++ wasn’t around when I started programming, so no it wasn’t the first language I learned. I started with BASIC. All home computers had some variety of BASIC built in, and it was a while before I realised there was anything else. Some programs were written in machine code, so I learnt that next — converting Z80 assembly language to hex codes by hand and typing them in as DATA statements of a BASIC loader program.

I don’t think I’ve programmed in anything that was really unusual, but I’ve used quite a few languages for various different systems. Programming a PIC with only 100 bytes of program space was a challenge, and I don’t suspect many people ever programmed the Psion Organizer II, though the programming language for it was similar to BASIC in many ways.

Can you recall any influential or maybe even outstanding books that helped establish yourself in the career?

Hmm. It would have to be the Z80 programming reference I had when I was around 10 or 11. I can’t remember the title, but I devoured that book. I knew every instruction by heart, including the hex opcodes and the clock counts. I found a similar guide to the 8086 on a shareware disk when we got a PC, printed it out and devoured that too.

These days my bookshelves have books like Design Patterns, Refactoring, and The Art of Computer Programming, though I still like to keep “close to the metal” with things like Intel’s Software Optimization Cookbook.

Now C++, and concurrency. Missing on opportunity for a few questions to the expert seems to be not such a good idea. So, to begin with, what don’t you like in the C++ 2011 concurrency support?

Good question! I don’t think there’s anything that I can point at and say “I don’t like the way that works”. I think it was a shame the “is_ready()” function got removed from future and shared_future, since it’s useful, and you can achieve the same goal with wait_for(seconds(0)) anyway.

You’re a maintainer of the Boost thread library. At the same time you develop your own library for concurrency in C++ called just::thread. What is special in this library?

Just::Thread is a strict implementation of the C++11 thread library, highly optimized for the specific platforms targetted, whereas Boost.Thread is aiming for portability, and doesn’t have the same interface or semantics in several cases. Though Boost.Thread is being brought further in line with C++11, it still lacks std::async, which is present in Just::Thread.

Just::Thread also has a special “checked” build which will identify the call chain leading to a deadlock should one occur.

Actors and the concept of sharing by communicating. It looks that the lack of this functionality doesn’t allows the concurrency in C++11 to shine in full power. Can you recommend any libraries providing such functionality?

The Just::thread Pro library currently in development will provide actors. There is also an example in my book of using message queues to create actor-style code.

I recall your saying that multi-threading in computations is tricky because it starts to make sense only after a certain volume. Do you have any recommendations on when it’s worth considering a concurrent and multi-threaded solution, and when it isn’t?

Yes, there’s an overhead to setting things up for multithreading. If your computation is quick enough then you may well be better off writing it single-threaded. Unless the multithreaded code is going to be clearly faster, then writing single-threaded code will save you a lot of headaches.

As ever, the key to optimizing for performance (and multithreading in this scenario is just that — an optimization) is to profile your application. Where is it spending its time? Can you parallelize that section? As Jason McGuiness demonstrated in his presentation at ACCU 2012, if you parallelize the wrong part of the code then it can be a lot of effort for no gain whatsoever.

Now, TDD. Where are you in this area? Have you embraced TDD and now use it everywhere? Is TDD applicable for developing concurrency libraries, for example, in C++?

I like TDD, and use it all the time. It encourages you to work in small steps, and the test suite you build up ensures that scenarios you’ve tested already don’t break as you address new scenarios.

It’s a bit harder to do TDD when developing concurrency libraries, but not impossible, and I still do it. The trick is in structuring your test to check the right thing. You typically need some form of barrier (I often use a std::promise with std::shared_future) so that you can set up all your threads in the right state, and then say “go!” The best way to test concurrent code is still to remove the concurrency — have well-defined communication mechanisms, and then test each thread in isolation.

Do you think that the code must be ideal, no trade-offs, no tolerance to inaccuracy or lack of the beauty? How do you judge your own code for being ready to release?

You should always /aim/ for perfect code. If you don’t, then it is very easy to end up with slap-dash unmaintainable code.

However, you don’t always get to write perfect code. Sometimes you can’t see a way to make it better, or the changes required are too extensive for the time frame you have. It is always better to have code that works correctly rather than code that looks pretty. If you have extensive tests then you can refactor your code later to a better structure without worrying about breaking it.

I judge code to be ready to release when I can’t think of a circumstance that would break it. If I can think of such a circumstance then I write a test for that, and fix it.

Can you name your three biggest “Never ever do that!” for programmers? and also for C++ programmers?

Tricky! “Never” is a strong word.

Here’s some things you should rarely do, in no particular order:

• Use global variables. Pass things you need as parameters or hold them as member variables rather than relying on globals, as globals make access patterns hard to understand.

• Use singletons. These are just global variables in disguise, and should be used with the same caution.

• Write multithreaded code without carefully thinking through the data access patterns from the separate threads. If you’re going to write multithreaded code, take the time to do it properly. You’ll save yourself a lot of effort in the long run.

For C++ specifically, you should rarely:

• Use malloc and free. This is C++, not C.

• Write code that requires you to use “delete”. If you have to use “new” then you should probably be using a smart pointer such as std::shared_ptr or std::unique_ptr to manage the memory. Often you’re better off using a container like std::vector to manage everything anyway.

• Overload operators in an unconventional manner. Sometimes it can be really useful (e.g. using << for stream insertion), but having a+b mean anything other than “adding” for the relevant data types is likely to yield confusion.

You wrote a book. Please, shed some light on this process. Why have you decided to write it in the first place, and how long did it take to prepare “C++ Concurrency in Action”? What was the most difficult part of it?

I wrote “C++ Concurrency in Action” because the opportunity came up, and I was one of those best-placed to write it, having been intimately involved in the drafting of the relevant sections of the C++11 standard.

It has taken 4 years to put it all together. One of the reasons it took so long is because the C++11 standard wasn’t finished when I started writing, and I had to add, remove and revise sections as the draft standard evolved, but even without that it was a lot of work. The hardest part was rewriting sections based on feedback from reviews.

Can you recommend other books about concurrency and multi-threading?

“Patterns for Parallel Programming” by Mattson, Sanders and Masingill is a good overview of how to structure parallel programs.

“The Art of Multiprocessor Programming” by Herlihy and Shavit is also good, but much lower level. This covers things like visibility, atomicity and consensus, and implementation of low-level structures such as queues, spin locks and monitors.

Now the question I ask everybody: sport programming and programming contests. Is it important to any developer to spend some time regularly solving puzzles? And have you even been involved in developing or implementing sophisticated algorithms?

I love puzzles. To an extent, programming is one big puzzle; that’s what makes it interesting.

I think puzzles use the same sort of thought processes as programming, so regularly doing puzzles can improve your programming. Practising anything helps make you better, so programming puzzles and contests are good for that — you have to break from your “normal” programming routine to focus yourself on something different, and try things out. I like the Intel Threading Challenge that they’ve been doing for a couple of years now. Even if you don’t enter, it can be good to do the challenges — they are fun, and really stretch you.

I did have a lot of fun once working on an algorithm for assisting helicopter maintenance engineers tune helicopters. They take a series of measurements from the helicopter with special equipment whilst it is flying in a series of different conditions, and then the software tells them what to adjust to make it fly better. Since helicopter flights are expensive, the goal was to tune the helicopter with the minimum possible set of test flights.

Are you an IDE or Vi/make guy? Do you think that it is still possible to develop sophisticated software simply in Vi, or all those days passed and considering proper IDEs is an inevitable step in enterprise software development?

I use emacs and make. I haven’t yet found an IDE that makes life easier for me. I think there’s a lot of potential in Eclipse/CDT, especially with the enhancements that Peter Sommerlad’s team are adding, but I haven’t found it enough to switch yet.

Do you think that “The Last Programming Language” exists? Or may it exist at all? Is it C++?

There is always scope for improvement in a programming language, and any given language makes some things easy to express and others hard, so I’m not sure that the “last” programming language could really exist.

If we ever get a sufficiently advanced AI, then we’ll end up using natural languages rather than specific programming languages, but I think that’s a long way off yet.

Let’s talk about hiring. If you need to hire a good C++ developer, what kind of questions you’d prefer you ask?

To some extent that depends on what I need from a new hire. If there’s time to train someone then specific C++ knowledge is not an issue, and I’d want to focus on things that highlighted general “developer aptitude”.

If I needed someone who already knew C++ then I’d want to ask questions that demonstrated that knowledge as well.

Wrapping up, do you have a list of, say, three things which each developer must implement through his or her career?

I’m not sure there are 3 things that every developer should implement. However, there are definitely things to be learnt from implementing basic data structures like lists and hash-maps, so I’ll take that as the first one.

For a second, I think a language parser can provide useful insight in putting together data structures, even if it’s a parser for a simple configuration file language rather than something complicated like C++. I think an arithmetic expression parser can be quite fun to write as a simple exercise.

Finally, I think it is worthwhile writing a client-server application. A web-app with Javascript code in the browser, and something running on the server would work for this, but anything where you are making calls from the client to the server over the network will do. The point is to do something where making a server call is considerably more expensive than making a local function call, so you have to consider the cost when designing the API. If there’s a UI then it can also be an exercise in handling backend latency whilst providing a “responsive” UI.

Thanks, Anthony. Looking forward for your new talks and books, hope for your future appearances on conferences.

■

// Anthony Williams, Alexander Demin

// July 2012

I suspect there will be series of blog posts about Raspberry Pi.

Frankly, even before receiving my own RPi I already had to solder the SD socket on another RPi which was broken. There is a rumor spreading that this type of SD sockets is very unreliable.

In contrast I cannot complain at all about an SD socket used in Maximite but it was a different type.

I decided to buy Olimex USB To Serial Cable. It was available on eBay as Olimex USB-SERIAL-CABLE Olinuxino console cable alternative to Raspberry Pi.

This cable provides 3.3V output levels which are strongly recommended for RPi (not 5V), and its 3 wires with female plugs can be directly connected to the RPi GPIO P1 header.

Unfortunately, at a time of this writing, there were no drivers for Mac OSX available on the official website, and OSX Lion (10.7.4) didn’t recognize this device. It was an unpleasant surprise because before I saw that Lion perfectly recognized other USB-RS232 hardware based on different chipsets (Microchip, for example).

I checked the Product ID and Manufacturer on the Olimex cable. When the cable is connected, click on “About This Mac > More Info… > System Report… > USB”.

Googling for a Prolific driver for Mac brought me to a great blog post, called “PL2303 Serial-USB on OSX Lion”. Following the instructions I installed the driver and connected to Raspberry Pi.

In short I did the following:

cd /tmp
wget http://xbsd.nl/~martijn/log/osx-pl2303.kext.tgz
tar xvzf http://xbsd.nl/~martijn/log/osx-pl2303.kext.tgz
cd osx-pl2303.kext
sudo cp -R osx-pl2303.kext /System/Library/Extensions/
cd /System/Library/Extensions
chmod -R 755 osx-pl2303.kext
chown -R root:wheel osx-pl2303.kext
cd /System/Library/Extensions

kextload ./osx-pl2303.kext
kextcache -system-cache

Note: Just in case, I made a copy of the “osx-pl2303.kext.tgz” file.

You can make sure that everything is installed correctly by:

kextstat -b nl.bjaelectronics.driver.PL2303

It should print the following:

Index Refs Address            Size       Wired      Name (Version) <Linked Against>
   74    0 0xffffff7f808ee000 0xb000     0xb000     nl.bjaelectronics.driver.PL2303 (1.0.0d1) <73 34 5 4 3>

Note: If you need to unload the driver for some reason, you check that the driver is loaded (by kextstat command above), and then:

sudo kextunload /System/Library/Extensions/osx-pl2303.kext

When the driver is installed, reconnect the cable to USB. There should be two device drivers created /dev/cu.PL2303-00002006 and /dev/tty.PL2303-00002006. You can check it by:

ls /dev/*PL*

It should print:

/dev/cu.PL2303-00002006     /dev/tty.PL2303-00002006

Now let’s connect the cable to the RPi header P1. It’s better to disconnect it from USB temporarily.

The pinout of the cable is the following:

An example of the connection:

When all three wires are attached, you can connect the cable back to USB. Now any terminal emulator software can talk to RPi via the /dev/cu.PL2303-00002006 device. I used Minicom:

minicom -D /dev/cu.PL2303-00002006 -b 115200

It should print something like this:

Welcome to minicom 2.6.1

OPTIONS: 
Compiled on May 10 2012, 07:16:49.
Port /dev/cu.PL2303-00002006

Press Meta-Z for help on special keys


Debian GNU/Linux 6.0 raspberrypi ttyAMA0

raspberrypi login:    

Finally, you may want to re-direct the RPi console from HDMI to the serial port (ttyAMA0). In this case RPi will be printing boot messages to the serial console instead of the monitor.

You need to login to your RPi (via SSH, telnet or just established serial connection) and execute the following:

sudo mount /dev/mmcblk0p1 /mnt

This will mount your boot partition to /mnt. Now you need to edit the kernel commund line:

sudo vi /mnt/cmdline.txt

In cmdline.txt make sure that console=ttyAMA0,115200 and kgdboc=ttyAMA0,115200, and there is no another console= assignment, for example to tty0. After you save the changes and reboot RPi, you may see RPi boot messages in the terminal emulator connected to /dev/cu.PL2303-00002006.

Now it is a museum of cryptography. Recently the Colossus was re-built, and now available to see it in action. My target was to see the Colossus and Enigma.

Today was a normal english day – raining all the way through. My lovely big brolly was incredibly useful.

If to skip a sticker on the windows, you may think that The War is still on.

Illustrations in the past were always created with love to details and general beauty.

A revolver. Just caught my eye because it was so close.

Frankly, I saw MP 38 (aka Schmeisser) so close first time in my life.

Do you recognized this tool? A twisted-pair crimp tool? This was from the 40th.

Looks like a key disc from Enigma-like machine.

In fact, the Enigma. To be exact – a bunch of them. I imagined the Enigma as a particular device which I saw in the American re-make of the “Das Boot” movie. But looks that even lazy people manufactured it.

A crypto device from Siemens.

Now a portable modification.

All exhibits are real and more than 70 years old.

The Bombe machine which was build particularly to crack the Enigma code.

The back view.

In action.

In a radio cabin.

On the way to The Colossus.

So, this is it – the top secret computer of the British secret service during the First World War. Recently it was re-built and now available in action.

This is a two panels stuffed with bulbs. The front panel, from the left.

The front panel, from the right.

The back view. It is all moving and cracking. The modern fans on the floor are cooling the bulbs.

The back view from the left.

Between the panels. The oscilloscope even displays some figures.

How Colossus worked.

Frankly, it wasn’t impressed at all. Yes, it is moving and cracking by tapes and spins, but… May be if they allow to code up a little program in place, I was much more excited.

In the same building with the Colossus, there is a The National Museum of Computing. Today it was, a shame, closed. Trying to bribe a person on duty to allow me to pop into mainframe and PC rooms did work out.

There is only a sad picture through the bars.

And this is the must have thing for any respectable bookstore (this is in the gift shop where they mostly sell books about the history of cryptography) – a sofa.

GMC-4 is a 4-bit microcomputer:

• 4-bit word
• 8 register, 1 flag
• 80 words in program memory
• 16 words in data memory
• 7 pre-loaded program (Organ, Sound Hit Game, Whack-a-mole, Tennis, Timer, Music Player, Morse Code Generator)
• 7 regular 2-pin LEDs
• one 7-segment LED
• a hex keypad
• a speaker
• hard reset button
• powered from three AA elements

The architecture is described in one small document – Programming the Gakken GMC-4 Microcomputer.

Yesterday my order arrived. GMC-4 is an add-on to an issue of a Japanese magazine called “Gakken”, #24, 2009. Unfortunately it’s all in Japanese language.

Funny that the back side of the box has a table of all GMC-4 instructions.

The board, speaker, case, parts of the keyboard.

The manual in Japanese is though.

Set up batteries and the speaker.

The board.

Sticking the keyboard.

Power on… It’s alive! Alive!!!

There is the keyboard, the 7-segment LED, the processor below it, then the “hard reset” button. There are seven independent LEDS on the top. They are used, for instance, to display the current address.

Let’s write something. For example, a program waiting for a key pressed and then displaying it on the 7-segment readout.

00: 0     KA 0       ; Read a key code (0-F) into A. If pressed, then Flag=0, else Flag=1.
01: F00   JUMP 00    ; If Flag=1 (not pressed), then go to 00.   
04: 1     AO         ; Out A to 7-segment readout and set Flag=1
05: F00   JUMP 00    ; If Flag=1 (it is always 1 here), then jump to 00.

To enter the code: RESET 0 INCR F INCR 0 INCR 0 INCR 1 INCR F INCR 0 INCR 0 INCR RESET

To run: RESET 1 RUN

Interestingly, this device allows step-by-step trace. If to run the program via RESET 6 RUN, it will be stopping after each instruction and displaying the current address on the top LEDs in binary code. To continue, we need to press INCR and so on. By pressing RESET we can interrupt the program and check out the registers (they are regular cells in memory). This is simple but convenient debugging.

Now slightly more complex example: a running light over the top LEDs displaying the number of the current one on the 7-segment readout.

00: 80    TIA 0     ; A=0
02: 1     AO        ; Out A to 7-segment readout.
03: 3     CY        ; Store A to Y.
04: E1    CAL SETR  ; Turn on an LED which number is in Y.
06: 84    TIA 4     ; A=4
08: EC    CAL TMR   ; Wait (A+1)*0.1 seconds (0.5s).
0A: E2    CAL RSTR  ; Turn off an LED which number is in Y.
0C: 3     CY        ; Restore A from Y.
0D: 91    AIA 1     ; A = A + 1
0F: C7    CIA 7     ; If A=7, then Flag=0. Otherwise Flag=1.
11: F02   JUMP 02   ; Go to 02, if Flag=1, and set Flag=1.
13: F00   JUMP 00   ; Unconditional jump to 00 (Flag=1 after the previous instruction).

Code: 8 0 1 3 E 1 8 4 E C E 2 3 9 1 C 7 F 0 2 F 0 0

Development tools

Everything is in Japanese, but Google Translate does an amazing job.

Simulator

The loader

Programming languages

• Assembler
• C compiler and Assembler
• BASIC

Wrap up

This is an amazing gadget. I started programming on Radio-86RK in machine code typing in the Monitor. Same here. Also, the issue of the magazine, which GMC-4 is bundled with, according to illustrations (unfortunately, I don’t read Japanese), is dedicated to the history of microprocessors starting from Intel 4004. Then there is a dozen of code examples and projects using GMC-4. Beautiful!

Let’s assume that:

• We compare C++ 2011 generating native code and vanilla Java 7 generating JVM code. The JVM code will be compiled into machine code by the JIT compiler later on at runtime when the code is executed first time.
• Suppose, both C++ and Java compilers generate the most effective code as language semantics in a maximum.

Denote A to be the linear machine code execution speed, B – the speed of compiling bytecode into machine code. So, the overall execution speed is:

V(C++) = A1 + B1
V(Java) = A2 + B2

Obviously, B1 = 0, because C++ generates native code and doesn’t require extra work at all at runtime. But B2 is always greater than zero by definition. Regardless how effective the JIT compiler is, it always requires some time to compile bytecode into native code. Furthermore, it doesn’t compile all at once. It compiles “just-in-time” when the code is actually executed. There is always a non-zero probability that the program execution flow may be paused by the JIT compiler. Even we assume that the JIT compiler uses sophisticated code prediction algorithms to reduce B2 as much as possible, it cannot eliminate B2 completely.

Now let’s have a look at A1 and A2. These parameters define how fast the generated code is. In my personal, totally subjective and biased view, C++ (not C) has more chances for better optimization because of the templates (the compiler has full language semantics to inline) and the native machine code generation (the compiler can use the most effective machine instructions in each situation). Unfortunately, I’m not an expert in Java generics. I have just heard that they are not “native” citizens in Java. Also the Java compiler has to generate portable bytecode and cannot optimize on each particular platform. I hope, the JIT compiler tries to optimize it at runtime but there is no semantics information available anymore. Then the JIT compiler faces the trade-off between the speed and quality of compilation. C++ doesn’t have this problem because the compilation phase can be as long as necessary.

Well, I cannot convince even myself that Java can be faster than C++. I’d like to, but I have no arguments.

With Stas we did a few straightforward benchmarks based on QuickSort. Java worked ~10% slower on average.

Prior to C++ 2011 we could say that C++ had no a well-defined memory model and a standard library for concurrency. It might allow Java to beat C++ using multi-threading and concurrency. But now C++ has these features. In general either C++ or Java provide pretty low-level API for concurrency comparing, for instance, to goroutines in Go or actors in Scala (though std::async() is quite powerful feature now in C++).

Agreed, 10% don’t always play the key role. Sometimes, advanced mechanisms of introspection, IDEs, managed execution, code hot swapping, etc. – in fact, everything what the Java platform provides, are more important than just byte threshing. But there is nothing about speed, right?

Lately I had an opportunity to play with a few devices of such sort, but slightly upper class based on ARM, not PIC32.

I apologize at once for quality of pictures, but I didn’t manage properly taking pictures of the bright computer screen neither on iPhone nor on a camera.

MK802 (aka Mini Android 4.0)

This is a device I have bought. It costs 70$ + delivery. There is another, more modern 1GB model already available on the official website. Mine has 512MB only.

My photos:

The USB #1. The device can be powered via this socket but they say it needs at least two amps, so powering from a regular computer may not work. I connected a keyboard here.

Mini HDMI. I also managed to connect to a DVI monitor via an adapter.

At the end face there is another USB port and an external power connector.

Micro-SD. The case of the device as a whole is a just bit bigger than a regular USB stick.

Connected to TV. Android is built-in. Booting time to this splash-screen is ~5s.

Time of the full boot is ~25s.

Connected a mouse, somehow entered a WiFi password in the virtual keyboard and checked out the proper website.

The desktop is optimized as a media center, that’s why everything is so big on the screen. After some fiddling with fonts, I had got this:

Now another proper website.

Internal storage characteristics.

The version of Android.

Pre-installed applications.

Finally, yet another proper website. Unfortunately, JSLinux wasn’t able to boot up fully. Maybe because of the web browser.

Built-in Android doesn’t require a flash card. If to prepare a card with another Linux, MK802 will boot from it. I have tried images of Ubuntu 12.04 and Lubuntu from the official website. Alas, it worked very slow and unresponsive. MK802 does require a proper distro optimized specifically for this platform.

On the official forum there are more photos of MK802. Also there are disassembly slides of a slightly different model.

Conclusion: It is not quite clear who is the audience of this device. Even Maximite allows to play with hardware, but here… Maybe running XBMC as a media center might have at least little sense.

CuBox

Produced SolidRun. The device was purchased by one of my colleagues, annoyed waiting his Raspberry PI (I’m also still waiting mine), and I was able to play with it.

CPU is slower, but the peripherals are much more advanced (no WiFi, though), and they also promise 1080p video. So CuBox is more likely to be a media center.

It’s a proper cube, which side is roughly as a match box.

Power, Ethernet, HDMI, Micro-SD, USBx2, eSATA.

SPDIF, micro-USB (console).

Frankly, the device looks being made in someone’s kitchen. On the second picture it’s clearly visible that the connectors are not quite aligned with the case. But the device in general works fine. It has Ubuntu 10.04 pre-installed.

Raspberry PI

This board was given to me by another colleague of mine who had eventually got it. It came as is, without a power supply and a flash card to boot from. You have to prepare everything by yourself. Fortunately, plenty of stuff is available online.

What does make PI different from MK802 and CuBox is general purpose ports (GPIO) (in the top left corner). Theoretically you can plug in your own homemade devices into Raspberry PI. PI was developed as an educational device, not an end user one.

Instead of the conclusion

The market of tiny computers is exploding. They grow as mushrooms after the rain. ARM is great, apart from one little issue – Chrome isn’t yet available on it due to obvious reasons.

#include <future>
#include <vector>
#include <deque>
#include <list>
#include <forward_list>
#include <typeinfo>
#include <iterator>
#include <iostream>

template <typename T>
void go(T f) {
  auto start = std::chrono::high_resolution_clock::now();
  f();
  auto stop = std::chrono::high_resolution_clock::now();
  auto duration = std::chrono::duration<double>(stop - start).count();
  std::cout << duration << "\n";
}

void erase(std::forward_list<int>& c, std::forward_list<int>::iterator i) {
  c.erase_after(i);
}

template <typename T> void erase(T& c, typename T::iterator i) {
  c.erase(i);
}

template <typename T> void test() {
  std::cout << typeid(T).name() << "\n";
  size_t const N = 100000;
  T v(N, 0);
  std::srand(0);
  for (auto t = N; t > 0; --t) {
    auto i = v.begin();
    std::advance(i, std::rand() % t);
    erase(v, i);
  }
}

int main() {
  go (test<std::vector<int>>);
  go (test<std::list<int>>);
  go (test<std::forward_list<int>>);
  go (test<std::deque<int>>);
}

The overloaded erase function is for uniformity of the test funtcion.

Compiling in VS11:

cl /O2 /EHsc test.cpp && test

Results:

class std::vector<int,class std::allocator<int> >
1.40678
class std::list<int,class std::allocator<int> >
8.85827
class std::forward_list<int,class std::allocator<int> >
8.70124
class std::deque<int,class std::allocator<int> >
9.19784

The difference isn’t huge but also not negligible at all as for list and forward_list.

A quote from the book:

I don’t believe anything really revolutionary has ever been invented by committee.

Huh!? Only for the sake of this quote this book is worth to be read.

Steve Jobs biography was interesting reading, even quite long though. Steve Wozniak’s book is much shorter, and also closer to people with engineering background.

Steve Wozniak tells a story how it all happened. How he spent nights at school time designing, re-designing, then again and again re-designing non-existent yet computers just on paper. It helped him develop a unique habit to always use a minimum number of chips in his circuit designs. A perfect example is a first floppy disk controller for the Apple 2 computer. Instead of a complicated and costly purely hardware solution he put a few chips and then implemented the rest in 6502 assembly language, having masterly fulfilled requirements for signal timings. Additionally, he designed, again using a few chips, a state machine, which he programmed directly in 4-bit machines codes also designed by himself (a handmade programmable logic matrix approach). Eventually, at some day Steve had invented the Apple 2, a revolutionary personal computer with the color TV-display and the full keyboard. By the way, contrary to the ordinary opinion, Jobs didn’t participate in its development. In a few years that computer made both Steves multi millionaires and started the Apple Computers. It is no wonder that the Apple surpassed everybody on the market – it was simpler, cheaper and have a bunch of unique features.

But that wasn’t the end of the story. After Steve got into a plan crash on a small airplane, operating himself, and after that incident he was suffering from partial amnesia for some time. Then he quit Apple, and, again, contrary to the ordinary opinion, he wasn’t fired, suspended or something because they all had quarreled. He just wanted to start another project – a universal remote control for consumer electronics. Formally, till now Steve is still an Apple employee, even with a salary.

But that wasn’t also the end of the story. For some reason Steve decided to organize an open air rock gig, a festival, similar to Woodstock. In fact, he did a few of them. Of course, the technology was involved. These festivals had a live satellite connection to USSR (they used equipment installed for the Olympic games in Moscow in the eightieth), right through the “iron curtain”.

After that Steve taught bases of computers at school for more than ten years and was engaged in philanthropy.

The whole slice of the book is also devoted to pranks Steve have been constantly doing through his life. For example, a Dial-A-Joke service he ran in his own apartment, or a portable TV jammer, and that big one, “The Zaltair Prank”, when Steve spread a rumor about a competing computer company, called Zaltair (remember The Altair which launched Bill Gates to the orbit?). Even Jobs believed it and only after years realised that it was a setup.

So, if you have at least a little interest in biographies, it is worth to read this engaging story about the good fellow, genius engineer, who transformed the industry of personal computers once and for all.

Seems I also have invented a pattern called “The biased attitude to your own product”. Here is a typical stream of thoughts when preparing a product release: “Oh, a little glitch here, a tiny typo there, this could be done better, that should be refactored”, etc. But when you open up a nicely packaged someone else’s product you think “Oh, man, how great and nicely polished it looks!”. But, of course, that “someone else” went through exactly the same thoughts “polishing” that product.

I’m regularly involved in producing releases, sorting bug reports, postmortem analysis, etc. I usually follow this kind of strategy: The sales grow? Yes. A number of customer issues decreases? Yes. A number of internal defects decreases? Yes. The conclusion: everything is fine, and we’re doing great. The rest is just a normal development process.

There is a “Disclaimer” link at the bottom of each post to avoid misunderstanding. I only refer to books which I read and reviewed by myself. It is not related in any way to publishers or authors.

Any information published commercially will be explicitly marked as advertising in the blog post content.

Any questions? Welcome to comments.

Related links

• Amazon affiliate program

When you have a class where you believe you need to write a copy constructor and copy-assignment operator you are doing something wrong. // Peter Sommerlad

Today my guest is Peter Sommerlad. I was lucky to attend his sessions at the ACCU 2012 conference recently. Peter is a C++ veteran and a contributor to the C++ standard. He is a professor at HSR Rapperswil, Switzerland, and director of IFS Institute for Software (http://ifs.hsr.ch). With a team of assistants and students he develops various tools and frameworks for C++ and other languages based on Eclipse.

Now we’ll have a chance to ask Peter a few questions, which could hide behind his talks and presentations but still be very interesting for the readers.

Hi Peter, thanks for the interview. I won’t be original here, but I cannot skip this question. What happened in the past that you turned to computers and become a programmer? Please, tell us your story.

I was very good at maths in school and in the last year I could take an additional course that included programming a programmable calculator, a HP 33 which had a stack for operands. In contrast to some of my friends who had a Casio, a TI-59 or even a Commodore V(I)C 20 home computer, I had not yet any idea of programming. However, with some explanations by our teacher I was able to “port” a TI-59 register-based lunar lander I found in a magazine to the HP 33 provided by the school. Once I got the hang of it, I “invented” a simple game: a ballistic shooting game. Goal was to hit a target at a random distance that was obstructed by a random generated wall in between. Something like Angry Birds but in in 1982 on a pocket calculator with a 10 digit seven segment LED display. You needed a lot of imagination to understand the game. You input 2 numbers: angle and velocity and the program would tell you if you hit the wall or by what distance you missed the goal. That was the first program I remember to have written myself.

That course on programming the HP 33, and the suggestion by my older sister who knew someone studying “Informatik” who earned good money while still a student made me drop my plans to sign up for a maths major and start out to study computer science (named Informatik in German) in Frankfurt/Main. Before I started university I got my first “real but home-” computer, a Sinclair ZX Spectrum, with enormous 48kBytes and cassette tape mass storage. That meant programming in BASIC where according to our maths teacher the “computer will assign the memory automatically to the named variables and you no longer needed to remember what you placed into register R1 and R2”. One of the first programs on my ZX Spectrum was drawing a red heart shape on the screen (yes, filling irregular shapes was a big problem then). I showed that my then girlfriend, but I am not sure she really appreciated that, since the computer got sometimes a bit more attention than she. I remember that, because from time to time I get reminded by her of that episode (we are now married for more than 20 years :-).

Then came university time and I had to learn assembly programming in the first semester on a Univac 1100. After the lecture on indirect addressing, one of my mates and me wrote a 2 statement program that created an endless indirect load to itself. Running it caused a “watchdog error” to occur after 2 seconds of CPU time (which was much on a mainframe then and meant all other programs were suspended for 2 seconds). After starting the program about 3 times, a guy from the operating team bursted into the computer room and shouted who was the one with our account. We slowly raised our hands and he took us to his office. After asking what we were doing and our explanation, he told us that he learned something. We had to promise never to run that program again and also never to tell that to our fellow students and got away with it. In the second semester came Pascal and now “real” programs. Pointers (in Pascal) were the first hurdle, because they were introduced without any application or motivation, such as a linked list. Only after that I was able to grok them. Some semesters later I started a job as a programmer in a small programming shop run by older fellow students. There I could actually apply my learning and got a lot of practice in solving real problems in a variety of languages, like dBaseII, Dataflex, UCSD Pascal, and Microsoft Pascal. For example, I implemented a B*-index based “data base” framework in Microsoft Pascal running on DOS.

I guess a very great impact was access to UNIX source code, especially the tools like make, awk, etc, during my diploma thesis project. I wrote a Modula-2 front end in C with lex and yacc. To be able to work at home, I had to port all the tools I needed on my 16 bit DOS computer. That required to understand the code’s inner working and structure and thus I learned from Dennis Ritchie and his colleagues a lot. It grew my taste for simple and elegant code.

Do you remember that “First Book” you read about computers and programming? And also what is your the most influential book across the entire career?

One of the first books on programming that I remember was Jensen/Wirth “Pascal User Manual and Report” and then Wirth’s “Algorithms and Data Structures = Programs”. However, the latter contained many bugs when the code was blindly typed in, for example, in the version I got, merge sort was not stable, so in principle that is a stable sorting algorithm.

However, there are so many books that influenced my thinking about programming it is hard for me to pick just one. But I can pick one that I co-authored: Pattern-oriented Software Architecture: A System of Patterns aka POSA1. First, while writing it I learned so much about software, architecture and teams. As well after publishing and while presenting parts of its material on conferences I met so many influencing people who clearly shaped me heavily. Some of them I call friends today.

I was excited about your saying at the ACCU, that before using some cool and sexy features of C++, I must be given a C++ driving license by you. I had found this idea to be very useful and practical for teaching C++ and also setting up company coding standards. But how do you split the language to “allowed and recommended to everybody” features and “off-license” ones?

That is a tough question to answer. I was referring to C++11 and many language features that people learned in the past are no longer that relevant for non-library programmers. But I’d like to give you an example.

Stroustrup’s “The C++ Programming Language” introduces a class Vector that manages its memory by a pointer as one of the examples for building classes. Writing such a class in a correct, exception safe way is very hard and even C++ experts make mistakes. A reader of that book might be tempted to build his own vector or string class (women are typically not that way) which is first wrong and second completely unnecessary. If put in his first project’s code and used throughout it also introduces a maintenance burden in the long run.

It is something like the situation with Wirth’s algorithms and data structures. “All” students who had a hard time learning that will program a linked list as the first thing in their first job, because they will “need” it anyway. While that might have been true in C, we now have much better data structures in libraries that belong to our languages (not only C++). I could start ranting about the Java library’s quality, but I leave that to my colleague who is teaching algorithms and data structures in Java at our university.

Now, after we learned what not to do, I recommend to use C++’s standard library, especially the algorithms, because loops is also what you learned and what is encapsulated there. With C++11 we no longer have to do resource management by hand, but we can use std::vector, std::string, std::shared_ptr/std::unique_ptr, etc, which will do all resource management for us and allows the compiler generated default constructors and destructors to be correct automatically. When you have a class where you believe you need to write a copy constructor and copy-assignment operator you are doing something wrong. With boost this is almost also always true in C++03.

To sum up, unless you have acquired a deep understanding on library implementation practices, use libraries instead of inventing your own. Even then, avoid unnecessary DIY resource management. After myself learning more and more about how hard it is to implement the standard library components correctly and portable, I greatly appreciate those who have the experience and patience to do so. That is actually close to rocket science. But typical problems to be solved by C++ do not require that, when you use the standard library and other libraries provided by theses rocket scientists. But remember, we are on the edge of tourist space travel.

You contributed to the C++ 2011 standard. What were you involved to?

Not much, because I was late in the game. However, I managed to obtain funding from our school thanks to our headmaster Prof. Hermann Mettler to host the C++ standardization committee meeting in Rapperswil in August 2010. Some sentences in the section about std::async have been (re-)written by me with the support from many others in the committee and some other mostly editorial corrections. However, while writing my book on Simple C++ I often have to refer to the standard document and find minor mistakes regularly that will influence a future version when fixed by the editor.

Is there something in C++ 2011 which you really hate?

How do you define hate? I can accept the standard like it is, because I know how it is actually produced. All the deficits that might annoy me are a target for fixing and working on the next version of the standard and I can work on that to do so. Sometimes you also have to accept the compromises the committee has to come up with.

If I would have been involved earlier and one of the features I championed would have been forcefully put down by my fellows I might be upset. May be, after thinking about that, I do not like that const is consistently put to the left, where logic tells me it should be put to the right. Both versions are syntactically allowed. I asked some people, if I should submit a paper to change that for a future version I was more or less forcefully held back by the elders.

Let’s talk about TDD. I know you developed a unit testing framework called CUTE. I believe, you did have some major reasons developing your own library. Why CUTE? Which interesting features does it provide?

First, it is header only and thus gives no hurdle such as linking to start with, especially for my students. I wrote an article on why I wrote it and it was published in an ACCU Overload magazine #75 (a slightly update version is available on http://wiki.hsr.ch/PeterSommerlad/wiki.cgi?CuTe). Main reason was that I was disappointed by CppUnit and also by another unit testing framework that I myself wrote for my work in the 1990s. Inspired by Kevlin Henney I tried to utilize the standard library and minimize the use of macros, even though, this meant, tests must be registered “manually” and aren’t “automagically” registered by static initializers (that GoogleTest will employ in its macros). I wanted to avoid relying on static initializers, because I have been bitten by them in the past with shared libraries and their inter-dependencies. The burden of test registration is mitigated by our CUTE Eclipse CDT plug-in (http://cute-test.com). The plug-in will generate the test registration code for you and even analyses if you forgot to register a test function and provides a quick fix doing so. Also nice is the diff viewer for failed ASSERT_EQUAL() checks, but that requires your values to be streamable on a std::ostream&. That provides a slight hurdle for embedded developers who want to run the testing framework on an embedded target, because of the sometimes prohibitive size of the iostream code. However, I plan to release a version of CUTE later this year that allows to configure if diff output should be provided or not, enabling a run on an embedded target that doesn’t allow iostreams.

Two of my students implemented the first TDD support for Eclipse CDT and it was productized by one of my assistants as part of our CUTE plug-in (http://cute-test.com). The TDD support will automatically generate variable definitions from using an unknown variable in code. It will furthermore generate class or enum types from using an unknown type in a declaration and it will also create (member-) function definition frames from calling an unknown function. This is very handy when either writing code in a test-driven or top-down design style. Soon the CUTE plug-in will integrate another student’s work: Mockator. Mockator generates code and build-system settings within Eclipse CDT to enable dependency injection in existing C and C++ code by so-called seams. Seams were introduced by Michael Feathers in his great book “Working Effectively with Legacy Code”. Based on these seams, Mockator can also generate test stubs and mocks for functions and types. You can read more about Mockator and its features in ACCU’s Overload magazine 108.

“IDE vs Vi” holy war. I know you actively use and enhance Eclipse CDT for easier unit testing and refactoring. Do you think that IDEs are the only future because of massively growing complexity of software over the years? or it is still possible just using Vi and “make” efficiently and quickly develop sophisticated software?

I know vi since about 1985. I still use it regularly to create simple files, like one-off shell scripts. I love it for being available everywhere.

However, for serious programming a good IDE is today the tool of the trade. It is like using a powerful car compared to a children tricycle to travel hundreds of miles. Yes, there are people who are fast on a tricycle, however, a car is much more comfortable. May be a car is a bad metaphor and an agile helicopter would be better, since a good IDE such as Eclipse CDT allows to quickly navigate code that it is very easy to understand an existing code base even if it is huge. Definitions and declarations are a mouse hover or a click away and the IDE usually keeps track where you come from, so you can easily get back. Any Java programmer who has used an IDE like Eclipse intensively will only consider programming in another language if similar IDE features are available. We are working hard to make Eclipse CDT at least as good for C++ than it is for Java. That is hard work and any financial sponsorship to our institute is welcome.

Do you believe that sport programming and programming contests should be a mandatory discipline for any student reading computer science?

In our university’s program we ask our students to perform a lot of lab exercises in programming. Also at least 3-4 major one term development projects are part of the bachelor degree. Doing lab exercises in a fun way can help students learn. For example, I often give exercises in C++ to solve a problem without explicit loops or recursion, only STL algorithms are allowed. A loop solution might be obvious, but figuring out a more elegant algorithm call helps a lot.

However, some contests are set up in a way that only speed is important and not the long term effects of clean code. That is one of the reasons I either provide or ask for unit tests in my lab exercises. I even provide unit tests in exams for code to be written by the students.

We usually learn by examples. Can you recommend where a beginner can look at the examples of very good modern code?

Such examples for C++11 are hard to find, since compilers aren’t around so long. In addition, many current C++11 example code is used to demonstrate the new features such as move semantics without really having a use case that is benefiting from that. I’ll try in my upcoming book to give good examples. Be aware, many C++ standard library implementations are much more complex than code you want to look at, because they have to cope with strange environments (macros), provide backward compatibility and need to compensate for compiler (version) deficiencies. Solving these problems are not typical for a beginner and give a wrong impression on how to write your own code. I’ll try to lead by example, but even CUTE has some warts, because I wanted to support C++03 with boost, std::tr1 and C++11.

What is the software development for you: craft or art? Do you think that good software must have beauty sources? Or if the requirements are satisfied, the tests passed, the coding standard is complied – the job is done, no art or beauty – just the result.

Software creation is art but also a craft. It requires thorough learning and practice to master. Most of art always requires also craft. They can not be separated.

I believe in code beauty. But that is relative, what is often good for beginners often suffers from the expert’s elegance. Knowing the tools available to you is essential to master them. Only knowing std::vector, for example, but not the STL’s algorithms will provide you working code, but not elegance, even if that is only counted by a function’s McCabe complexity (by the way another C++ CDT plug-in currently developed by students).

We were the first ones after Bill Opdyke to try to implement C++ refactoring. We co-created large parts of CDT’s internals to support refactoring and code generation based on ASTs (abstract syntax tree). So running the tests is not enough, code must be cleanly refactored. I do a lot of code reviews, and I often see chances to simplify it. But I doubt that talent can be automated, it is a mixture of taste and experience. It requires not only seeing the smells but also their removal. On the other hand, if you look at Martin Fowler’s smells and refactorings they often come in contrasting pairs.

Some coding standards even promote what I consider bad practice and introduce unnecessary complexity. For example, Google suggests that “out”-parameters for functions should be implemented as pointers, even in C++. The motivation behind that is, that at the call site one would need to write foo(&var) to pass the address of the variable deliberately. This is thought to improve code readability. However, while necessary in C due to the lack of references, this guideline just imposes C problems solved by C++ references onto C++ code. Within such a function, one must check that the passed in pointer is not nullptr (to use a C++11 term) and one never can be sure it is actually valid if the caller made a mistake, or one could ignore the problem at all and crash when a user passes nullptr. A C++ solution should declare the function to take a reference instead, like void foo(type & var); and an IDE will show that when you hover on the call site, without the additional need of marking it in the code by an ampersand. Using the language features effectively to write less and simpler code should be the guideline and not the protection of programmers sticking with legacy editors and not knowing their tool.

A coding standard or any rule is only good if it can actually be enforced. Tools like (PC-)lint help to do that. Often such classic tools are command-line and build-process based. More modern versions are integrated in an IDE and provide not only analysis but also quick-fixes for the found problems, for example, FindBugs for Java or FXCop for C#. Configuring them to avoid false positives without missing real problems is always an effort. To ease the application and use of lint, we created Linticator an Eclipse CDT plug-in that will automatically configure lint and run it and visualize its results within the IDE (http://linticator.com).

Can you name three the biggest “no-no” for programmers in any area?

Being too self confident, instead do write (Unit) Tests and run them regularly. Any current software without automated tests is unprofessional in my view.

Lack of architecture or architectural knowledge is one of the big problems I often encounter when doing code reviews. This is often fostered by lack of understanding of good design.

A simple last thing: global variables also in the disguise as singletons. Pass in all dependencies from the outside as arguments. This will also show you, where you have too much coupling. Know the patterns to deal with that and writing unit tests helps you to avoid that early.

Which programming language would you recommend as a very first one for beginners?

There are so many to choose from. I do not want to recommend one, because each will have its deficits. It is more important to learn more than one and those with different paradigms such as functional, object oriented, dynamically typed, statically typed, compiled, interpreted, virtual machine-based or machine language translated. An old saying is “you can write FORTRAN code in any language”. This should not be done with multiple languages, but you need to learn and apply the language’s idioms consciously.

Today there are plenty of educational resources available online. You can even be officially graded online. Do you believe that on-site education is still important?

Yes. Interaction with students is so important. It is not only them getting feedback, but it also allows me to learn from them and improve my teaching. Both kinds of feedback are important and are more efficient when immediately. A remote learning environment can not provide that bi-directional feedback, but it can be a great help for those who can not be there for some reasons. But I believe it is very expensive to provide a good remote learning environment.

In the martial arts they have the concept of schools, or styles, and they compete trying to prove the excellence. Is there something similar to the software development?

I know of people to use martial arts metaphors, such as coding dojos. I have no idea about martial arts, the only sport I like, except regular cardio-vascular exercise, is skiing. This is not a team sport but allows me to get into flow and forget everything. Intensive work can be like that. But software is a team sport and regular exercises of standard moves, aka, code katas can help to shape your subconsciousness in taste for good code and style, but only if you either have that already or get the feedback to develop that. A problem I see is, that many teachers of programming novices lack the (current) practical experience of programming (including myself, but at least I try to be exposed to code regularly).

Hiring. If you need to hire a good C++ developer, how would you do that? The language is very complicated, so how would you asses a good developer?

I always have asked for code examples written by the person. Once I declined a very good PhD for exactly the reason that his code was badly written and “smelly”.

Today, also the willingness to adapt to change and architectural taste is important. A candidate must understand Patterns and must know there are more than 23 GOF patterns. And better even understand that some of them like Singleton are better not employed in a software’s design.

Well, wrapping up. The final question: do you think in software development that we choose our profession, or the profession chooses us?

I started out by choosing my university subject. However, the great thing about software is that we are able to invent and create our own tools. It is like if a baker would be able to create a new oven or dough mixer by baking it. So after learning a bit of it, I actually got chosen by my profession. I am doing it for about 30 years now, teaching it for about 25 years (I started teaching programming while still university student), writing articles and books about it since about 18 years and still love doing all that. I have as my goal to make this world a better world for programmers and thus hope to rid our world from bad software. I understand that I will not achieve that, but it gives me a lot of direction.

■

Thanks Peter for you time. Looking forward for your new software and talks.

// May 16, 2012

// Peter Sommerlad, Alexander Demin

#include <string>
#include <cstdio>

int main() {
  std::string s = "12345678";
  std::printf("[%s]\n", s);
}

This is an obvious typo of missing s.c_str() instead of just s. Weirdly, Visual Studio doesn’t warn you even with /Wall. More over, the code works!. But gcc warns though:

warning: cannot pass objects of non-POD type 'struct std::string' through '...'; call will abort at runtime

and the program dies with Illegal instruction.

Did they really implement a trick in STL to make programs having such typo working just because this typo is very common?

So, let’s go right to the comparison table from the official web site.

I wanted to figure out unique features for the sake of what they had started this project. I checked out a presentation from OSCON 2011.

#1

#2

#3

Here is my subjective fifty cents:

• Clear “no” to GPL (a stone into git, hg, bzr). They say that “Not-GPL” is more attractive for enterprise use.
• Ability to use external data storage for the repository (for example, MSSQL or Oracle). Again, large companies may like it.
• File locks (a controversial feature, in my view)
• Bug tracking (fossil also can do this).
• Build tracking (seems to be a really unique feature).
• Written in pure C, and it natively supports Windows, Linux and Mac from the day one.

There are a few principle decisions:

• No ‘index’ (as in git) and it will never be there.
• There is no way to change the history, at all (for example, even a typo in a last commit cannot be fixed).

Personally for me, these two “features” definitely don’t fit into my way of using DVCS. For this reason I have stopped using fossil on my personal projects, though I loved the ability to have Wiki and a bug tracker built into the repository.

As a result it turns out that the key features of Veracity are “Not-GPL” and the external storages. It is obviously an attack on the corporate market. By the way, the company founder, Eric Sink, already had a DVCS product and company now acquired by Microsoft.

So, my subjective conclusion is that this is an attempt to bring DVCS into corporations.

Extra fifty cents

Eric Sink wrote a book, “Version Control by Example”, in which he gives more or less fair comparison of Veracity against a few flagship DCVS. I flipped through it in half an hour, and I found a super quote, which I furiously share and actively promote (the selection in bold is mine).

11. Don’t comment out code

When using a VCS, you shouldn’t comment out a big section of code simply because you think you might need it someday. Just delete it. The previous version of the file is still in your version control history, so you can always get it back if and when you need it. This practice is particularly important for web developers, where the commented-out stuff may adversely affect your page load times.

Never use a get prefix for getters.

Why?

Usually we consider member functions to be getters if they simply return a reference or pointer to a private member, just for the sake of data incapsulation. Such functions don’t do anything really, they have no payload.

So, there is nothing requiring a “get” verb to describe its meaning. The getter is just an alias of a member. The “get” prefix is only required when a function does a real data transformation or computation, for instance, getLastTick() or getFullUserName(). But it is even better here to replace “get” by something more meaningful, extractLastTick() and buildFullUserName().

■

Related posts:

• Naming convension for getters and setters in C++

You can find plenty of options online to organize your standing desk. Here is my solution. It appeared by accident. When we rented an apartment, there was no room for a proper computer place in the kitchen (yes, we do need a computer place there, for sure), and my wife discovered this nice cheap table and a bar stool at IKEA.

BJÖRKUDDEN

HENRIKSDAL

We used it for a while as a hotspot for quick googling or ordering food online. After some time at found myself spending more and more time at this place. It was very convenient for some reason. Normally you stand of feet, and sometimes, for example, during a long “wave period”, you can place your bottom at the high chair. After we moved, I used a regular computer workplace for a while at my office at home, but then switched to the standing desk. This is how it looks:

By the way, its legs are perfect for fitting socket extensions and other stuff.

And the previous, classical workplace on the right:

P.S. By the way, I believe that people who invented a pull-out keyboard shelf for computer tables never worked at the computer. It’s hard to imagine more inconvenient way of using the keyboard. Also I personally never use any office chair if I cannot sit, resting a foot under myself.

Leonard Richardson, Sam Ruby, “RESTful Web Services”

So, skipping a tedious chapter of HTTL libraries in Ruby, Python, Java and curl, more tedious chapter on HTML4, XHTML, HTML5, Atom, XML, a little less tedious chapter about a couple of Ajax libraries, extremely tedious chapter with standard HTTP response codes, unrealistically tedious chapter about standard HTTP headers, the essence of the book can be expressed very briefly. This is my summary of the entire book.

Developing a web-service you should follow some recommendations:

1. (Main one) Designing a URL structure you put as much meaning into there as possible. You should treat URLs as a query language for your data, which is readable and writable by human. For example, instead of http://domain/engine.php?func=123&id=test for get user details it should be something like http://domain/users/test.
2. Use all standard HTTP commands and response codes: not only GET and POST, but also PUT, DELETE, OPTION, HEAD, and not only 200 and 500, but the range of the standard HTTP response codes.
3. HTTP response should contain links to other resources reachable from the current one.

That’s it! Now you know the whole book.

This is a rare situation when I’m seriously considering money back. Unfortunately I cannot re-sell because it’s a bloody e-book.

Yesterday I was in mood and decided to install RetroBSD there.

RetroBSD is a real UNIX, derived from 2.11BSD and intended for embedded systems with fixed memory mapping. Its target platform is Microchip PIC32 microcontroller with 128KB RAM and 512KB of Flash, which can provide flexible RAM partitioning between user and kernel modes.

RetroBSD implements real preemptive multitasking and memory protection using hardware capabilities. It provides standard POSIX API (fork, exec, wait4, etc.), plus it may have a C compiler in the system allowing on-board development. The kernel is burned into the chip and the filesystem is loaded from an SD-card.

RetroBSD works not only on Maximite, but also on a few alternative boards based on PIC32 (chipKIT Max32, Sparkfun UBW32, Microchip Explorer 16, Microchip PIC32 USB/Ethernet Starter Kit, Olimex Duinomite, Duinomite-Mini and Duinomite-Mega, eflightworks).

After a little fiddling around with the bootloader and library dependencies I managed to build the binaries and upload them onto the device.

A solemn moment of powering on…

This is UNIX!

Of course, games first. The Worm.

Canfield

Now a bit of programming – Forth.

At the moment RetroBSD only communicates via the serial port and doesn’t support VGA and PS/2 interfaces provided by Maximite, but Serge Vakulenko, the author, has their support in the roadmap.

Without synchronization

#include <future>
#include <iostream>

volatile int value = 0;

int loop(bool inc, int limit) {
  std::cout << "Started " << inc << " " << limit << std::endl;
  for (int i = 0; i < limit; ++i) {
    if (inc) { 
      ++value;
    } else {
      --value;
    }
  }
  return 0;
}

int main() {
  auto f = std::async(std::launch::async, std::bind(loop, true, 20000000));
  loop(false, 10000000);
  f.wait();
  std::cout << value << std::endl;
}

Compiling via clang:

clang++ -std=c++11 -stdlib=libc++ -O3 -o test test.cpp && time ./test

Run:

SSttaarrtteedd  10  2100000000000000

11177087

real    0m0.070s
user    0m0.089s
sys 0m0.002s

Obviously, the increment and decrement operations aren’t atomic, and the value variable contains garbage at the end.

LOCK

#include <future>
#include <iostream>

volatile int value = 0;

int loop(bool inc, int limit) {
  std::cout << "Started " << inc << " " << limit << std::endl;
  for (int i = 0; i < limit; ++i) {
    if (inc) { 
      asm("LOCK");
      ++value;
    } else {
      asm("LOCK");
      --value;
    }
  }
  return 0;
}

int main() {
  auto f = std::async(std::launch::async, std::bind(loop, true, 20000000));
  loop(false, 10000000);
  f.wait();
  std::cout << value << std::endl;
} 

Run:

SSttaarrtteedd  10  2000000100000000

10000000

real    0m0.481s
user    0m0.779s
sys 0m0.005s

Now value has the correct value at the end, but, of course, this is a dirty non-portable hack for x86 only. For example, this code works for me only if compiled with -O3. Otherwise it crashes with “illegal instruction” because the compiler injects extra stuff between the LOCK instruction and the following increment or decrement.

Atomic

#include <future>
#include <iostream>
#include "boost/interprocess/detail/atomic.hpp"

using namespace boost::interprocess::ipcdetail;

volatile boost::uint32_t value = 0;

int loop(bool inc, int limit) {
  std::cout << "Started " << inc << " " << limit << std::endl;
  for (int i = 0; i < limit; ++i) {
    if (inc) { 
      atomic_inc32(&value);
    } else {
      atomic_dec32(&value);
    }
  }
  return 0;
}

int main() {
  auto f = std::async(std::launch::async, std::bind(loop, true, 20000000));
  loop(false, 10000000);
  f.wait();
  std::cout << atomic_read32(&value) << std::endl;
}

Run:

SSttaarrtteedd  10  2100000000000000

10000000

real    0m0.457s
user    0m0.734s
sys 0m0.004s

The result is correct, and the time is almost the same as with LOCK. Not surprisingly, atomic under cover also uses LOCK but in a portable and guaranteed way.

Spinlock

#include <future>
#include <iostream>
#include "boost/smart_ptr/detail/spinlock.hpp"

boost::detail::spinlock lock;
volatile int value = 0;

int loop(bool inc, int limit) {
  std::cout << "Started " << inc << " " << limit << std::endl;
  for (int i = 0; i < limit; ++i) {
    std::lock_guard<boost::detail::spinlock> guard(lock);
    if (inc) { 
      ++value;
    } else {
      --value;
    }
  }
  return 0;
}

int main() {
  auto f = std::async(std::launch::async, std::bind(loop, true, 20000000));
  loop(false, 10000000);
  f.wait();
  std::cout << value << std::endl;
}

Run:

SSttaarrtteedd  10  2100000000000000

10000000

real    0m0.541s
user    0m0.675s
sys 0m0.089s

A bit slower but not much.

Mutex

#include <future>
#include <iostream>

std::mutex mutex;
volatile int value = 0;

int loop(bool inc, int limit) {
  std::cout << "Started " << inc << " " << limit << std::endl;
  for (int i = 0; i < limit; ++i) {
    std::lock_guard<std::mutex> guard(mutex);
    if (inc) { 
      ++value;
    } else {
      --value;
    }
  }
  return 0;
}

int main() {
  auto f = std::async(std::launch::async, std::bind(loop, true, 20000000));
  loop(false, 10000000);
  f.wait();
  std::cout << value << std::endl;
}

Run:

SSttaarrtteedd  10  2010000000000000

10000000

real    0m25.229s
user    0m7.011s
sys 0m22.667s

Now it works much slower, really.

Benchmark

Of course, the result depends really on the platform and the compiler (I tested on Mac Air and clang). But for me it was quite interesting to see that spinlock, in spite of its more sophisticated implementation comparing to atomics, works not much slower.

Sadly, my clang 3.1 still doesn’t support atomic, and I had to use boost.

A.h:

#include <memory>
class A_pimpl;
class A {
  A();
  ~A();
  std::unique_ptr<A_pimpl> p;
}

I believed to some reason that it will not work because the class A_pimpl is partially defined. I was surprised as a child when tried and realised that it works! The fact #1 is not equivalent to the fact #2.

Now in A.cpp I can write:

#include "module.h"
#include "pimpl.h"

A::A() : p(new A_pimpl()) {}
A::~A() {}

Everything above also works for std::shared_ptr (C++ 2011), boost::scoped_ptr, and boost::shared_ptr.

Update

It is important that class A must an explicitly provided destructor. Moreover, its body must in A.cpp, not in the header. Otherwise, it will be a compilation error, for example, “error C2338: can’t delete an incomplete type”.

His presentation is available with examples.

Unfortunately, I missed his second presentation, “Dataflow, Actors and High Level Structures in Concurrent Applications”. In fact, it was about implementing the concept of actors, similar as in Erlang, in C++ 2011.

This presentation is also available with examples.

Recommend.

Going further. Anthony’s book, “C++ Concurrency in Action”, has been released recently.

If you want an exhaustive explanation about the C++ memory model, please read the chapters 5 (along with all others).

#include <future>

int main(int argc, char* argv[]) {
  for (auto i = 0L; i < 1000000; ++i) {
    auto f = std::async([](){ return 0; });
    f.get();
  }
  return 0;
}

This code consistently crashes (and doesn’t throw). When decreasing a number of iteration, at some point to stops crashing.

The compiler cl.exe 17.00.40825.2, IDE 11.0.40825.2 PREREL. Posted on Stack Overflow. They say, likely it is the bug.

Where to file a bug to VS?

This is my naive approach:

int naive_quick_sort(std::vector<Type>::iterator begin, std::vector<Type>::iterator end) {
  auto const sz = end - begin;
  if (sz <= 1) return 0;

  auto pivot = begin + sz/2;
  auto const pivot_v = *pivot;

  std::swap(*pivot, *(end - 1));
  auto p = std::partition(begin, end, [&](const Type& a) { return a < pivot_v; } );
  std::swap(*p, *(end - 1));

  if (sz > 4096) {
    auto left = std::async(std::launch::async, [&]() {
      return naive_quick_sort(begin, p);
    });
    naive_quick_sort(p + 1, end);
  } else {
    naive_quick_sort(begin, p);
    naive_quick_sort(p + 1, end);
  }
  return 0;
}

void quick_sort(std::vector<Type>& arr) {
  naive_quick_sort(arr.begin(), arr.end());
}

Everything is simple, but it’s worth to consider some details. This is a threshold, 4096, determining when we switch the parallelism off. Why 4096? Just a guess, without a particular explanation.

Benchmark

The are three candidates:

• the naive implementation (using async)
• the same naive implementation, but single threaded (replace if (sz > 4096) to if (false))
• std::sort() (replace naive_quick_sort(arr.begin(), arr.end()) to std::sort(arr.begin(), arr.end()))

We sort an array of 50000000 elements of the type int64 (signed) with 10 runs and take an average. The values are random:

std::tr1::uniform_int<Type> uniform(
  std::numeric_limits<Type>::min(),
  std::numeric_limits<Type>::max());
std::mt19937_64 engine;

void generate(std::vector<Type>& v) {
  std::for_each(v.begin(), v.end(), [](Type& i) { i = uniform(engine); });
}

Don’t ask me, why I convert the data between big and little endians back and fourth. It was implemented to compare this implementation with another one, in Java. In measurements we count only the “pure” time.

The compiler is VS 2011, 64-bit. CPU Intel Core i5 2.53GHz, 4 cores.

Iteration  With async()   One thread   std::sort()
--------- --------------- ------------ ------------
 1         2512            6555         7309
 2         2337            6320         6977
 3         2450            6516         7180
 4         2372            6388         6933
 5         2387            7074         7189
 6         2339            7399         7040
 7         2434            6875         7040
 8         2562            7060         7187
 9         2470            7050         7145
10         2422            6846         6898
--------- --------------- ------------ ------------    
Среднее    2428.5          6808.3       7089.8

The time is in milliseconds.

It turns out that the naive implementation based on async() is three times faster then the single threaded std::sort(). A weird slowdown of the std::sort() against my naive single thread version can be explained, perhaps, that the data I generated is good, and my naive implementation is just lucky.

Any practical use out of it at all? Probably not. It is difficult to predict the behaviour of the implementation on different sets of data. For example, how to choose the threshold properly? Is it worth to start using thread pools?

The full source is below, including the generator.

Build and generate the data:

call "%VS110COMNTOOLS%..\..\VC\vcvarsall.bat" amd64 && ^
cl /Ox /DWIN32 sort_async.cpp && ^
sort_async generate

Warning! It generates 8GB of data.

Build and test:

call "%VS110COMNTOOLS%..\..\VC\vcvarsall.bat" amd64 && ^
cl /Ox /DWIN32 sort_async.cpp && ^
sort_async

File sort_async.cpp:

#include <vector>                      
#include <iostream>
#include <fstream>
#include <sstream>
#include <algorithm>
#include <iomanip>
#include <future>
#include <random>
#include <chrono>
#include <cstdlib>

const int ITERATIONS_NUM = 10;
const int DATA_SIZE = 50000000;

typedef __int64 Type;

inline void endian_swap(Type& x) {
  x =
    (0x00000000000000FF & (x >> 56))
  | (0x000000000000FF00 & (x >> 40))
  | (0x0000000000FF0000 & (x >> 24))
  | (0x00000000FF000000 & (x >>  8))
  | (0x000000FF00000000 & (x <<  8))
  | (0x0000FF0000000000 & (x << 24))
  | (0x00FF000000000000 & (x << 40))
  | (0xFF00000000000000 & (x << 56));
}

std::tr1::uniform_int<Type> uniform(
  std::numeric_limits<Type>::min(),
  std::numeric_limits<Type>::max());
std::mt19937_64 engine;

void generate(std::vector<Type>& v) {
  std::for_each(v.begin(), v.end(), [](Type& i) { i = uniform(engine); });
}

void check_sorted(const std::vector<Type>& v, const std::string& msg) {
  for (auto i = 0; i < v.size() - 1; ++i) {
    if (v[i] > v[i + 1]) {
      std::cout << "\nUnsorted: " << msg << "\n";
      std::cout << "\n" << i << "\n";
      std::cout << v[i] << " " << v[i + 1] << "\n";
      std::exit(1);
    }
  }
}

std::string data_file_name(const int i, const std::string& suffix) {
  std::ostringstream fmt;
  fmt << "trash_for_sort_" << i << suffix << ".bin";
  return fmt.str();
}

void save_file(std::vector<Type> array, const std::string& name) {
  std::for_each(array.begin(), array.end(), [](Type& i) { endian_swap(i); });
  std::ofstream os(name.c_str(), std::ios::binary|std::ios::out);
  auto const bytes_to_write = array.size() * sizeof(array[0]);
  std::cout << "Saving " << array.size() << " bytes to " << name << "\n";
  os.write((char *)&array[0], bytes_to_write);
}

int main_generate(int argc, char* argv[]) {
  std::cout << "Generation\n";
  for (auto i = 0; i < ITERATIONS_NUM; ++i) {
    std::vector<Type> unsorted(DATA_SIZE);
    generate(unsorted);
    save_file(unsorted, data_file_name(i, ""));
    std::cout << "Sorting...\n";
    std::sort(unsorted.begin(), unsorted.end());
    check_sorted(unsorted, "check sorted array");
    save_file(unsorted, data_file_name(i, "_sorted"));
  }
  return 0;
}

void load_file(std::vector<Type>& array, const std::string& name) {
  std::cout << "Loading " << name;
  array.resize(DATA_SIZE, 0);

  std::ifstream is(name.c_str(), std::ios::binary|std::ios::in);
  auto const to_load = array.size() * sizeof(array[0]);
  is.read((char *)&array[0], to_load);
  if (is.gcount() != to_load) {
    std::cerr << ", Bad file " << name
      << ", loaded " << is.gcount() << " words but should be " << to_load << "\n";
    std::exit(1);
  }
  std::for_each(array.begin(), array.end(), [](Type& v){ endian_swap(v); });
}

int naive_quick_sort(std::vector<Type>::iterator begin, std::vector<Type>::iterator end) {
  auto const sz = end - begin;
  if (sz <= 1) return 0;

  auto pivot = begin + sz/2;
  auto const pivot_v = *pivot;

  std::swap(*pivot, *(end - 1));
  auto p = std::partition(begin, end, [&](const Type& a) { return a < pivot_v; } );
  std::swap(*p, *(end - 1));

  if (sz > 4096) {
    auto left = std::async(std::launch::async, [&]() {
      return naive_quick_sort(begin, p);
    });
    naive_quick_sort(p + 1, end);
  } else {
    naive_quick_sort(begin, p);
    naive_quick_sort(p + 1, end);
  }
  return 0;
}

void quick_sort(std::vector<Type>& arr) {
  naive_quick_sort(arr.begin(), arr.end());
}

int main(int argc, char* argv[]) {
  if (argc == 2 && !std::strcmp(argv[1], "generate"))
    return main_generate(argc, argv);

  std::vector<double> times;
  auto times_sum = 0.0;
  for (auto i = 0; i < ITERATIONS_NUM; ++i) {
    std::vector<Type> unsorted;
    load_file(unsorted, data_file_name(i, ""));

    std::vector<Type> verify;
    std::cout << ", ";
    load_file(verify, data_file_name(i, "_sorted"));
    check_sorted(verify, "verify array");

    std::cout << ", Started";
    auto start = std::chrono::high_resolution_clock::now();

    quick_sort(unsorted);

    auto stop = std::chrono::high_resolution_clock::now();
    std::cout << ", Stopped, ";
    auto duration = std::chrono::duration<double>(stop - start).count();
    std::cout << duration;

    check_sorted(unsorted, "sorted array");

    const auto match = unsorted == verify;
    std::cout << (match ? ", OK" : ", DON'T MATCH");

    times.push_back(duration);
    times_sum += duration;

    std::cout << "\n";
  }

  auto const average = times_sum / ITERATIONS_NUM;
  auto const max_element = *std::max_element(times.begin(), times.end());
  auto const min_element = *std::min_element(times.begin(), times.end());
  auto const average_fixed = (times_sum - max_element - min_element) /
                             (ITERATIONS_NUM - 2);

  std::cout << "Average: " << average << "s, " 
            << "Average without max/min: "
            << average_fixed << "s." << std::endl;
}

Finally, a CPU utilization graph. We see 100% spikes on each iteration.

Update

There is an article from Microsort, “Dynamic Task Parallelism”, also showing how to implement multithreaded Quicksort.

func format_time(t time.Time) string {
  return t.Format("2006.01.02-15.04.05")  // Similar to YYYY.MM.DD-hh.mm.ss
}

A full list of “magic” values (from time/format.go):

stdLongMonth      = "January"
stdMonth          = "Jan"
stdNumMonth       = "1"
stdZeroMonth      = "01"
stdLongWeekDay    = "Monday"
stdWeekDay        = "Mon"
stdDay            = "2"
stdUnderDay       = "_2"
stdZeroDay        = "02"
stdHour           = "15"
stdHour12         = "3"
stdZeroHour12     = "03"
stdMinute         = "4"
stdZeroMinute     = "04"
stdSecond         = "5"
stdZeroSecond     = "05"
stdLongYear       = "2006"
stdYear           = "06"
stdPM             = "PM"
stdpm             = "pm"
stdTZ             = "MST"
stdISO8601TZ      = "Z0700"  // prints Z for UTC
stdISO8601ColonTZ = "Z07:00" // prints Z for UTC
stdNumTZ          = "-0700"  // always numeric
stdNumShortTZ     = "-07"    // always numeric
stdNumColonTZ     = "-07:00" // always numeric

A few examples:

ANSIC       = "Mon Jan _2 15:04:05 2006"
UnixDate    = "Mon Jan _2 15:04:05 MST 2006"
RubyDate    = "Mon Jan 02 15:04:05 -0700 2006"
RFC822      = "02 Jan 06 15:04 MST"
RFC822Z     = "02 Jan 06 15:04 -0700" // RFC822 with numeric zone
RFC850      = "Monday, 02-Jan-06 15:04:05 MST"
RFC1123     = "Mon, 02 Jan 2006 15:04:05 MST"
RFC1123Z    = "Mon, 02 Jan 2006 15:04:05 -0700" // RFC1123 with numeric zone
RFC3339     = "2006-01-02T15:04:05Z07:00"
RFC3339Nano = "2006-01-02T15:04:05.999999999Z07:00"
Kitchen     = "3:04PM"
// Handy time stamps.
Stamp      = "Jan _2 15:04:05"
StampMilli = "Jan _2 15:04:05.000"
StampMicro = "Jan _2 15:04:05.000000"
StampNano  = "Jan _2 15:04:05.000000000"

I didn’t see such approach before, I’ve found it clearer and more logical.

A code with std::endl:

#include <string>
#include <iostream>

int main() {
  for (int i = 0; i < 1000000; ++i) {
    std::string s(1, 'x');
    std::cout << s << std::endl;
  }
  return 0;
}

Compile and run:

clang++ -o endl -O3 endl.cpp && time ./endl >rubbish

real    0m4.518s
user    0m1.080s
sys 0m3.311s

A code with \n:

#include <string>
#include <iostream>

int main() {
  for (int i = 0; i < 1000000; ++i) {
    std::string s(1, 'x');
    std::cout << s << '\n';
  }
  return 0;
}

Compile and run:

clang++ -o endl -O3 endl.cpp && time ./endl >rubbish

real    0m0.263s
user    0m0.236s
sys 0m0.008s    

The difference is obvious. The second one is much faster.

std::endl always flushes the stream. In turn, \n simply puts a new line character to the stream, and in most cases this is exactly what we need. And std::flush, if necessary, can be called afterwards, once, explicitly.

Yesterday the book had arrived. I wanted to write a proper detailed review, but realised it doesn’t make sense at all. This is a must have book if you’re interested in the new C++ 2011. The book is an exhaustive reference with a lot of examples of the new STL and the language itself. In my view, this is the first big C++ 2011 book at the moment.

It is called live coding. What You Code Is What You See - WYCIWYS.

I thought his demos were hardcoded. But it turns out, there is a real programming language allowing such crazy things in runtime. Check out a short video:

The language is called circa. Currently it is alpha. On my amateurish opinion in the game development, it is very convenient for experiements with gameplay, at least for 2D games.

Here is an interesting article from the author about preserving the runtime state on live code changes.

A collection is videos with more live coding.

For instance, a mini project having one file, and you don’t want to drag the Google Test or cmockery in. I usually do something like this:

#include <cassert>

void foo(...) {
  // something
}

...

#ifdef UNIT_TESTING
void Test_for_a_particular_use_case() {
  // Initialization
  ...
  assert(condition_1);
  ...
  assert(condition_N);
}
...
int main(...) {
  Test_for_a_particular_use_case();
  std::cout << “All tests passed.” << std::endl;
  return 0;
}

#else

int main(...) {
  // a proper main
}

#endif

Of course, you still can forget to add your test to main(), and it seems working, but the test simply doesn’t run.

But! All this is trifles in comparison with the advantages which the tests provide.

P.S.

By the way, I’ve noticed an interesting habit: when I’m writing a function working with files, I always create two:

void FunctionDoingSomethingFromStream(std::istream* is) {
  ...
}

void FunctionDoingSomethingFromFile(const std::string& filename) {
  std::ifstream is(filename);
  return FunctionDoingSomethingFromStream(&is);  
}

The first function is perfect for testing because you can pass in a mocked std::istringstream. The second doesn’t really need to be tested due to its simplicity. It may be tested though in QA, but not in unit testing during within build.

There is another interesting trick from Kevlin Henny. Usually a subject of testing (a class or a function) has a pre-condition and a post-condition. It’s worth to highlight these stages in comments by “Given”, “When” and “Then” words. For example:

void Test_for_a_particular_use_case_to_check() {
  // Given:
  ClassToTest a;
  // When:
  a.do_this(...);
  a.do_that(...);
  a.setup_something(...);
  // Then:
  assert(condition_1);
  ...
  assert(condition_N);
}

Such explicit split enforces proper test structuring: without loop and branches, and testing for only one scenario (for another scenario it will be another test).

■

For example, my PS1 variable is:

\W `vcprompt -f "%m%u %s:%b"`\$

making the prompt in bash to be like this:

_engine +? git:master$

It says that the current directory is _engine and there is a git repository here, the current branch is master, and there are changed (+) and also untracked files (?). If there is no repo in the current repository, the prompt will be normal.

vcprompt supports not only git, but bzr, cvs, darcs, fossil, hg, svn.

It requires Python, and is also hardly usable on Windows.

Because of laziness I began using Blogspot. It gives plenty of bells and whistles: templates, widgets, instant indexing by Google, statistics, even comments become to be a tree at some point. Everything seems to be great, but alas, the online Blogspot editor isn’t designed to create programmer posts. When it requires to insert a code or a table, sufferings begin. For example, for my another blog, not about programming, called “Boiled eggs, sir!”, “the capabilities” of Blogspot are more or less enough.

I want to keep sources of the posts in a nice and clear format, not spoiled by HTML. It turned out that the posts sources are spread across my computer here and there, sometimes in multiple copies. You begin creating a post simply in an editor only formatting paragraphs, without links and images, and eventually you save the final version. Then you compose HTML during that the post body maybe also changed. Thus I’m lazy to update the original, non-HTML version, so finally, only the spoiled by HTML version remains up to date. But this is not the end of the store still. Often after publishing you spot on a typo, go to the Blogspot online editor and fix it right on the page. Again, that very first document and its local, prior to uploading the HTML version, don’t have all the corrections. Ultimately, only the online edition in Blogspot is relevant. Of course, it is possible to export the entire blog from Blogspot (maybe even automatically, on regular basis) and back them up, but again, everything will be in the HTML format.

Some time ago I began using ReST. The life had become a bit easier. ReST allows to write text in the more or less predictable layout (paragraphs, links, code), and then HTML is generated from it, and then later pasted (again, manually) into the Blogspot editor. All my attempts using Google Command Line, in fact, failed. The problem of the outdated original documents (after online fixed) still stood. Furthermore, ReST didn’t solve the problem with pictures. I had to upload them somewhere in advance to do the a proper local preview.

I almost cannot explain why, but the idea of dynamic blog engines like WordPress scared me. Use of a database to keep posts seemed to me unnecessarily complicated.

I was almost ready to choose an intermediate solution – Doku Wiki, for example, vak.ru. This engine is dynamic, but it stores the posts as files. Plus, it supports versioning. Doku can be used as the engine for the whole site, not only the blog. Though its design is unsightly, but pictures and other arbitrary attachments are handled by Doku automatically.

There was another option, which I almost signed up for – TiddlyWiki. TiddlyWiki is my favorite tool on Windows for notes. Why only Windows? Because on Mac I simply use regular plain text files in Documents or Desktop, and Spotlight provides the instant search. Therefore on Mac the instant search, the killer feature of TiddlyWiki, didn’t make any sense. But I digressed.

It turns out, there are TiddlyWiki fans which converted it to a blog engine, a static-dynamic mutant.

For instance, here is an example of a blog running TiddlyWiki – Rich Signell’s Work Log. Esoterics, in my opinion. It isn’t quite clear even how to implement comments, at least Disqus. But surprisingly, there is public hosting based on Tiddly – tiddlyspot.

Finally, I had got hooked on purely static blog engines. The charm of it is that such blog can be hosted anywhere. Neither database nor even no server scripting is required. But further – it is more. GitHub and Heroku provide not only free hosting but also git as the CMS.

For example, there is a static blog engine called Jekyll. In Jekyll posts are created in Markdown or Textile, therefore the formatting issue is solved. No ugly HTML anymore. In fact, the engine can drive the entire website, not only the blog, plus some source files are deployed as a blog.

The comments, as the main dynamics in blogs, can be implemented via, for instance, Disqus. By the way, there is a quite popular pure zen approach to the comments – static ones (to me, such word combination is the oxymoron in the first place). With the static comments there is a comment entering form somewhere on the page along with the statically rendered existing comments. You enter a comment and submit. Then it gets sent to the blog owner. He or she approves (or rejects) it, clicks somewhere, and the comment is placed to a file. Then the blog is re-compiled and finally deployed to public. Obviously, it is far from real-time, which is the heart of commenting, in my humble opinion.

I very much appreciate discussion, and such approach is not for me. I use Disqus. By the way, all the comments can be easily exported from Disqus, and, for instance, converted to a static form.

But let’s go back to Jekyll. GitHub directly supports it (its author is the GitHub co-founder) and can render Jekyll projects for you (of course, you still can render locally). You simply git push your stuff to GitHub, and after a moment it becomes visible in GitHub Pages.

Heroku works slightly different. Heroku hosts Ruby, therefore a Heroku project is a bunch of pages, plus a mini web server application serving them. Looks a bit scary, but such server in Ruby is very simple:

require 'bundler/setup'
require 'sinatra/base'

class SinatraStaticServer < Sinatra::Base  

  get(/.+/) do
    send_sinatra_file(request.path) {404}
  end

  def send_sinatra_file(path, &missing_file_block)
    file_path = File.join(File.dirname(__FILE__), 'public',  path)
    file_path = File.join(file_path, 'index.html') unless file_path =~ /\.[a-z]+$/i  
    File.exist?(file_path) ? send_file(file_path) : missing_file_block.call
  end
end

run SinatraStaticServer

As strangely enough it sounds, Heroku as blog hosting is simpler than GitHub. Additionally, your git repository on Heroku remains private, whereas on GitHub by default it is public. I don’t see any problems with it though, because it is all available anyway to everybody via the web site, but some people prefer keeping blog guts or bits of unfinished work in private.

Also, both GitHub Pages and Heroku support a proper second level domain if you have one.

Well, I had chosen Jekyll and Heroku hosting. Alas, vanilla Jekyll has no HTML templates or themes, and you have to design it. If you’re lazy, you can try Octopress.

Octopress is a static blog engine based on Jekyll. “Based” means, this is Jekyll plus a bunch of useful plugins, a very nice HTML5-compatible theme, and scripts helping in deployment to GitHub Pages and Heroku.

So, I had installed Octopress, tried a couple of posts, rendered locally, deployed to GutHub and Heroky. Everything was amazing, I had began the most tedious part – converting posts from lovely Blogspot. In fact, it was almost pure manual cut-and-paste work. Three weeks of struggle, and my poor three hundred posts were done.

Everything was ready to launch my nice or shiny static blog. But here I had met my main disappointment. The precious Jekyll, written in Ruby, rendered my hundred posts (attention!) 15 minutes (on Mac Air). As you can understand, at the beginning I had to rebuild a hundred of times – design changes, formatting etc. So, 15 minutes wasn’t even close to any acceptable time, at all.

At random, I found a bottle neck in Jekyll/Octopress – the majority of those 15 minutes were spent generating the RSS file, atom.xml. For some reason, in original templates, that RSS filed included only the latest twenty posts. But my blog is small, and I included all posts there. It had caused that 15 minutes build.

All this seemed to me an absurdity (at all my love to Ruby). After some analysis (at that time I began to understand how Jekyll worked, more or less) and unwillingness to fix Jekyll trying to speed it up, I asked a question – what if I develop the static engine by myself? using Jekyll’s principles? After all it is only work with files, strings, and templates. Besides, I wanted to blog in two languages, but Jekyll didn’t support that. Having my own engine seemed to be a very cool idea because I would free to implement exactly that I wanted.

Instrumentation? As a real man in C++/boost? Perhaps. I suspect it would work very fast, but it is boring. I’d chosen Go. It is natively compiled, has simplified memory management (thanks to the garbage collector), regular expressions, maps, templates, Markdown library. Everything, except the later, is out of the box. There should not be any performance issues. Here just the first release of Go had been arrived, now installation on both Windows and Mac became much easier.

So, after three nights, my own wheel had been made – Goblog. The project is fully open. The web site and its source are in the same repository.

Architecture

There are two main locations: a project and a target web site/blog. The first contains source files. In the process of assembly the project files are copied to the target directory preserving the relative directory structure. By default, the files are copied “as is”, as binaries. If a file has one of the following extensions: html, xml, or js, it is processed by Go template library. Files with the markdown extension are also processed by the Markdown library.

Directories:

• <root> – Here is the compiled/built web site as it is visible at http://demin.ws/.
• <root>/_engine – This is the project: source files and the blog generator.

Subdirectories and files in _engine:

• _includes – Include files available by {{include “filename”}} macro.

• _layouts – Layouts (see below).

• _site – Actually, the source directories and files. This directory is the root of a future site. Files from here are copied to the target directory. Some files are processed by templates.

• _posts – Posts. These files are process specially. Besides templates, they are renamed according to the blog structure, where the date is included to URL: domain/blog/year/month/day/post-title.

The posts are Markdown files having a special preamble and name. These files are deployed to a separate directory, /blog. The post meta information is collected and propagated to the template place holders. Also the post sources are used to build the reverted index of search.

Layouts

The concept of layouts is grabbed from Jekyll. If a post or a page has the layout attribute in its preamble (for example), a layout of the given name is loaded (from _layouts directory), the post body is put to the Page.child place holder, the layout is rendered, and the result becomes the final HTML page of the post. This is incredibly useful for similar pages, like posts. The layouts can be nested.

Generator

Now, the generator – main.go.

To build the blog, I go to the _engine directory and do:

make

It prints roughly the following:

_engine$ make
gofmt -w=true -tabs=false -tabwidth=2 main.go
go run main.go 
Go static blog generator  Copyright (C) 2012 by Alexander Demin
Words in russian index: 18452
Words in english index: 3563
15.672979s
Processed 344 posts.

If no errors, in the root directory (in .. with regard to _engine) the compiled files should be created, ready for deployment. On my Mac Air the build takes 15 seconds (hello, Jekyll/Octopress and goodbye). The entire project is under git, so it is clear anytime which files are new, deleted, or changed.

Then the built site can be viewed locally (see below).

When you are happy with changes, let’s add and submit them (the sources from _site/ along with the assembled files) locally:

git add ../*
git commit -m "New post about ..."

At last, let’s deploy to GitHub:

git push

Almost immediately the files become visible at demin.ws.

In Makefile there is a couple of extra commands making life easier.

Local testing

To check/preview the site locally I temporarily add 127.0.0.1 demin.ws to /etc/hosts and launch a mini web server. Remember, how it looked in Ruby? Tiny, right? Check out this:

package main
import "net/http"
func main() {
  panic(http.ListenAndServe(":80", http.FileServer(http.Dir(".."))))
}

Nice? Now run it:

go run server.go&

It may require sudo to acquire the port 80.

Generally speaking, there is no need touching /etc/hosts, just use localhost:80. Only for checking the RSS file, atom.xml, having full URLs, you have to fiddle with hosts.

Syntax highlighting

As a Markdown extension I have a special tag for blocks of code:

{% codeblock lang:xxx %}
...
{% endcodeblock %}

I inherited the idea from Octopress. Markdown library already has the syntax for code:

``` xxx
...
```

where xxx is a language.

My own tag aims to add additional attributes easier, for example, turning on the line number, tabs conversion, etc.

Then I had to decide how to highlight the syntax. I tried a couple of online libraries, which color right on the page, but all of them had minor glitches here and there, and I went for static coloring.

The first, which came to my mind, was pygments. It seemed okay, but thanks to the Python, it worked really slow. The total built time had increased from 15 seconds to two minutes. The main time was spent for a code coloring. I started thinking about a cache for the already highlighted code and similar nonsense, but after a little research the problem was solved radically.

I just needed a colorizer, implemented in a language suitable for this problem. I found two options: GNU Source-highlight and Highlight. Both are in C++, the both worked almost instantly.

For example, a guy compared pygments vs syntax-highlight.

I liked Highlight because it supports more languages (for example, the first one even didn’t support Go). After migration to Highlight, the built time came to normal, ~15-16 seconds, and I was satisfied.

The colorizer is invoked from the regular expression callback parsing the {% codeblock %} tag (function highlight()).

Markdown editors

There are plenty of editors having Markdown preview. I use MarkdownPad for Windows and Marked on Mac.

Tag (categories)

At the moment I have decided not to use tags at all. Based on a personal experience, I have realised that I never use tag neither in my blog nor in others. Besides, views on the post categorization changes over time, and at times you have to maintain a completely outdated tag just for the sake of uniformity. For example, what is the point of the c++ tag in my blog? Who on Earth even used it?

But the minimalism isn’t a way to make life more complicated. On the contrary, I search in my old posts all the time. With Blogspot I went to the front page and used ⌘-F (sorry, CTRL-F) to search across the post titles. Exactly for this purpose I started placing almost all informative posts into the right column on the front page.

In the new blog everything “works” similar with the post catalog on the front page. During the migration I had improved some titles to be more informative for search.

But! All of this doen’t matter anymore because of the fully functional context search capability in the new blog.

Sanity checks

One of annoying inconveniences of Jekyll is that it never checks anything in the content. But I passed though it fully converting posts from Blogspot: broken links, missing quotes, bad dates and so on. That’s why the Goblog checks everything – formats, links, overall semantics, and it stops the build in the case of an error. When I added the check_links() function, I had found a bunch of dead links.

Two languages

There was another issue with Jekyll – an absence of bilingualism. I want at least two languages in the blog, bit I didn’t want to hard code the “transparent” support of Russian and English. So, I added the concept of the language to each post (or a file), and the language attribute is propagated to templates. Goblog doesn’t know about particular languages, but it allows to figure it out in templates.

For example, the front page in Russian and the front page in English.

Minor issues

I don’t like web programming: javascript, css, html, or web design, which I cannot do at all. But I had to do it for Goblog (Octopress was much easier in this). I based my design on the Jekyll’s author site. The design is very simple. Besides, many people read the blog via RSS anyway and go the blog web site for commenting only. Hence, I need the properly working RSS feed and the post page providing comfortable reading, without fancy fonts and weird formatting.

Conclusion

Do you think, I’ll start convincing you to use my engine? Not at all. I tried to make it flexible and not hard coded specifically to this blog, but I had to migrate my old posts from Blogspot and comments, support two languages etc. At the result there are a couple of places still hard coded (especially in the area of Disqus links).

I can only recommend trying to implement the personal blog engine by yourself. Why? First, it usually takes a few nights only. Second, you will implement the only required bared minimum without complicated and unnecessary bell and whistles. Third, it is fun.

■

Related links:

• GNU Source-highlight
• Highlight
• Adding Color to the Console: Pygments vs. Source-highlight

The first edition of this book was in 1982. Old code in C, which was considered to be “good” in the past, today quite often doesn’t stand up to criticism (even though its criticising is a pointless exercise), but it’s worth to study.

I was extremely curious to have a look at puzzles in C created in eighteenth. So, fasten your seat belts. If The Hell in C exists, this is it. I’ve given a few “problems” from this book.

Pointers

Preprocessor

Storage classes

For me, these snippets are obvious candidates for Obfuscated C Code Context. When have to deal with such code in real life, switching to another reality is the the best solution.

After skimming the book to the end (they give the solutions there), you begin to understand its usefulness. Without seeing bad code, it’s difficult to recognize the good one.

Just have come across this book in Twitter. A simple search gives plenty of options to read it.

I wouldn’t buy it for myself because for a more or less experienced programmer it doesn’t give anything new. But if you have a hour or two, it makes sense, for a tea or a coffee, to browse and refresh in memory various quirks of pointers, memory models (thanks to the weird segmentation inherited from 8086 and 80286), ABI, peculiarities of linking in UNIX and Windows, and interesting facts from the history of the languagemand its editions over time.

This book has a little bit of everything.

P.S. By the way, I thought I know C much better than C++, just because the language is smaller and simpler. More over, I began programming in C much earlier. Thus I’d taken a Brainbench C test at some time, and it turned out that there are still many aspects of the language which I haven’t come across so far.

Agreed, the Brainbench is only a test, in many respects far from reality, but nevertheless allowing to expand the scope of self-understanding.

From clarity perspective the best way is:

std::vector<int> create_vector(const size_t N) {
  std::vector<int> v;
  v.resize(N, 0xDEADC0DE);
  return v;
}

Here the vector instance is being returned by value, which means potential deep copying of the object to the context of the caller.

Immediately a question raises: if the vector is huge, such deep copying can be expensive but absolutely redundant. A “more clever” approach could be:

void create_vector(const size_t N, std::vector<int>* v) {
  v->resize(N, 0xDEADC0DE);
}

Here the vector is passed by pointer which guarantees the absence of deep copying. But this code looks quite bad.

Let’s compare the performance of these two approaches on the 100MB vector of ints. The compiler:

Apple clang version 3.1 (tags/Apple/clang-318.0.45) (based on LLVM 3.1svn)
Target: x86_64-apple-darwin11.3.0

By value:

#include <iostream>
#include <vector>

std::vector<int> __attribute__((noinline))
create_vector(const size_t N) {
  std::cout << "by value" << std::endl;
  std::vector<int> v;
  v.resize(N, 0xDEADC0DE);
  return v;
}

int main(int argc, char* argv[]) {
  for (size_t i = 0; i < 10; ++i) {
    const size_t N = 1024 * 1024 * 100;
    std::vector<int> v = create_vector(N);
    if (v[i] != 0xDEADC0DE) {
      std::cout << "Test is rubbish" << std::endl;
      return 0;
    }
  } 
  return 0;
}

Run:

clang++ -O3 -o by_value by_value.cpp && time ./by_value

Result:

0m4.933s

Now by pointer:

#include <iostream>
#include <vector>

void __attribute__((noinline))
create_vector(const size_t N, std::vector<int>* v) {
  std::cout << "by pointer" << std::endl;
  v->resize(N, 0xDEADC0DE);
}

int main(int argc, char* argv[]) {
  for (size_t i = 0; i < 10; ++i) {
    const size_t N = 1024 * 1024 * 100;
    std::vector<int> v;
    create_vector(N, &v);
    if (v[i] != 0xDEADC0DE) {
      std::cout << "Test is rubbish" << std::endl;
      return 0;
    }
  } 
  return 0;
}

Run:

clang++ -O3 -o by_pointer by_pointer.cpp && time ./by_pointer

Result:

0m4.852s

Time is both cases is the same. It turns out that it’s worth to choose the first one. It looks nicer and cleaner, and it’s as fast as “by pointer”.

There are two explanations. The first, probably the main one, is RVO, Return value optimization. In RVO the compiler figures out that the local instance of the vector is supposed to be a return value, and it allocates it directly in the caller context. So it doesn’t need to copy it afterwards. In fact, the compiler implements passing by reference, but it does it implicitly, without spoiling the beauty of the source code. This trick works for any classes, not only from STL.

But the optimization isn’t guaranteed thing, and there is one more tool. The standard STL containers are implemented such way that even on deep copying they only copy by value a small control structure, and the payload is “moved” by flipping pointers. It’s small overhead, but perhaps it’s worthwhile for the sake of preserving the code clarity.

Also in the context of move semantics in C++11 there won’t be unnecessary copying at all if the class is “properly” implemented (which is true for STL classes).

Conclusion: use STL containers where ever possible and trust the compiler. Sometimes, of course, the compiler is wrong, but such cases are much less than vice versa.

There are plenty of books published about writing code. Thus I was skeptical after a friend of mine showed me yet another one. To my surprise, without a long introduction it went straight to the point and coined the following statements right in the first chapter:

• Code should be easy to understand.
• What make code “better”?
• The fundamental theorem of readability.
• Code should be written to minimise the time it would take for someone else to understand it.
• Is smaller always better?
• It is better to clean and precise that to be cute.

The style of the book was precise and concrete. I felt that in less than two hundred of pages it’s quite ambitious to cover the subject of “the art” and decided to order the book to find out how they’re going to do that.

Eventually, a few hours of reading on weekend were worthy. Though an experienced programmer will barely find any eye openings in the book, but surprisingly, this is a compact, concise and solid handout for juniors. Without too much theory, always using real examples, the authors go through many key points of writing code: how to name variables, functions and classes, how to structure the code, how to deal with efficiency-readability trade off, how to comment, where to compromise and where be a perfectionist. Again, it’s all in less then two hundred pages. Plus they briefly touched unit testing.

The authors not only tell you what is good and bad, they always demonstrate “why” on examples gradually improving “regular” code. At the end they put a real example of a class counting network traffic and returning a number of bytes transferred in the last hour and day.

They began with a naive implementation and then worked through two more versions showing that sensitive balance between efficiency and readability. I think that even experienced developers may find this example interesting to play with.

To sum up, this book can fit perfectly into your team book shelf and be used as a quick reference of how-tos. Buying for yourself is questionable, because at home you’d probably prefer something more fundamental.

It could be fixed this way:

void f(int a, int /* b */) {
  ...
}

But it looks ugly.

There is a better way:

#define DISCARD_UNUNSED_PARAMETER(x) (void)x

void f(int a, int b) {
  DISCARD_UNUNSED_PARAMETER(b);
  ...
}

This macro is clear, and you can easily find all such places in the project.

By they way, in Go this situation is treated as the error, not warning. It can be annoying when you add, remove, comment out and put back things all the time, because sometimes you do have unused stuff. But eventually such approach doesn’t allow that temporary garbage to spread and stay in the code (for example, in C++ who on Earth wants to clean up the list of included STL headers after hours of coding?)

<script language="javascript">
(function() {
    function async_load(){
        var s = document.createElement('script');
        s.type = 'text/javascript';
        s.async = true;
        s.src = 'URL of a script to load';
        var x = document.getElementsByTagName('script')[0];
        x.parentNode.insertBefore(s, x);
    }
    if (window.attachEvent)
        window.attachEvent('onload', async_load);
    else
        window.addEventListener('load', async_load, false);
})();
</script>