putenv()
function.
We rolled out this class into QA, which works on various platforms (AIX,
HP-UX, Solaris, Linux and Windows). Everything seemed to be okay, unit tests
passed and the production code didn’t crash. Alas, QA machines controlled by
Hudson/Jenkins are usually overloaded, and quite often it causes very
unexpected issues. After a week we discovered that sometimes the code crashed
on AIX when calling std::system()
. Even more, it crashed inside this
function, that was for sure. In most cases any “strange” or “magical” behaviour
is related to memory problems. truss, injected into command lines passed
to system()
, showed that some environment variables in the child process
had corrupted values.
We began investigating the new class, EnvironmentVariablesManager
. Below
is its simplified version but still containing a tricky bug. You may try
finding the issue by yourself first, and then read further down.
The simple main()
function below easily reproduces the issue.
#include <vector> #include <map> #include <string> #include <unistd.h> class EnvironmentVariablesManager { public: typedef std::vector<char> VariableContainer; typedef std::map<std::string, VariableContainer> Variables; void put(const std::string& name, const std::string& value) { VariableContainer pair; PairToContainer(name, value, &pair); const std::pair<Variables::iterator, bool> inserted = vars_.insert(std::make_pair(name, pair)); if (!inserted.second) inserted.first->second = pair; putenv(&inserted.first->second[0]); } private: void PairToContainer(const std::string& name, const std::string& value, VariableContainer* pair) const { pair->clear(); std::copy(name.begin(), name.end(), std::back_inserter(*pair)); pair->push_back('='); std::copy(value.begin(), value.end(), std::back_inserter(*pair)); pair->push_back('\0'); } Variables vars_; }; int main() { EnvironmentVariablesManager env; env.put("DB2_HOME", "a"); env.put("DB2_HOME", "12345678"); }
Valgrind complains on this code saying that putenv
tries reading some memory
after freeing (this trace is from OSX).
clang++ -o putenv_check putenv_test.cpp && valgrind ./putenv_check
==1046== Memcheck, a memory error detector
==1046== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==1046== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==1046== Command: ./putenv_check
==1046==
--1046-- ./putenv_check:
--1046-- dSYM directory is missing; consider using --dsymutil=yes
==1046== Invalid read of size 1
==1046== at 0x2A8A3B: __findenv (in /usr/lib/system/libsystem_c.dylib)
==1046== by 0x232C62: __setenv (in /usr/lib/system/libsystem_c.dylib)
==1046== by 0x216A7E: putenv (in /usr/lib/system/libsystem_c.dylib)
==1046== by 0x100001999: EnvironmentVariablesManager::put(std::string const&, std::string const&) (in ./putenv_check)
==1046== by 0x1000015DE: main (in ./putenv_check)
==1046== Address 0x100012560 is 0 bytes inside a block of size 11 free'd
==1046== at 0x563A: free (in /usr/local/Cellar/valgrind/3.8.1/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==1046== by 0x10000208C: __gnu_cxx::new_allocator<char>::deallocate(char*, unsigned long) (in ./putenv_check)
==1046== by 0x10000201D: std::_Vector_base<char, std::allocator<char> >::_M_deallocate(char*, unsigned long) (in ./putenv_check)
==1046== by 0x100002483: std::vector<char, std::allocator<char> >::operator=(std::vector<char, std::allocator<char> > const&) (in ./putenv_check)
==1046== by 0x1000018A8: EnvironmentVariablesManager::put(std::string const&, std::string const&) (in ./putenv_check)
==1046== by 0x1000015DE: main (in ./putenv_check)
==1046==
==1046==
==1046== HEAP SUMMARY:
==1046== in use at exit: 2,425 bytes in 34 blocks
==1046== total heap usage: 58 allocs, 24 frees, 2,824 bytes allocated
==1046==
==1046== LEAK SUMMARY:
==1046== definitely lost: 18 bytes in 1 blocks
==1046== indirectly lost: 0 bytes in 0 blocks
==1046== possibly lost: 0 bytes in 0 blocks
==1046== still reachable: 2,407 bytes in 33 blocks
==1046== suppressed: 0 bytes in 0 blocks
==1046== Rerun with --leak-check=full to see details of leaked memory
==1046==
==1046== For counts of detected and suppressed errors, rerun with: -v
==1046== ERROR SUMMARY: 9 errors from 1 contexts (suppressed: 1 from 1)
To me it was not that obvious what was wrong. For example, if to remove the
second call env.put("DB2_HOME", "12345678")
, the issue disappeared
(valgrind didn’t complain any more). So, we started to suspect the following
lines of code:
if (!inserted.second) inserted.first->second = pair; putenv(&inserted.first->second[0]);
If to change the code little bit (if fact, doing exactly the same operation but in a slightly different way):
if (!inserted.second) inserted.first->second.assign(pair.begin(), pair.end());
the error report changes:
--1087-- ./putenv_check:
--1087-- dSYM directory is missing; consider using --dsymutil=yes
==1087== Invalid read of size 1
==1087== at 0x2A8A3B: __findenv (in /usr/lib/system/libsystem_c.dylib)
==1087== by 0x232C62: __setenv (in /usr/lib/system/libsystem_c.dylib)
==1087== by 0x216A7E: putenv (in /usr/lib/system/libsystem_c.dylib)
==1087== by 0x1000016A9: EnvironmentVariablesManager::put(std::string const&, std::string const&) (in ./putenv_check)
==1087== by 0x10000129E: main (in ./putenv_check)
==1087== Address 0x100013560 is 0 bytes inside a block of size 11 free'd
==1087== at 0x563A: free (in /usr/local/Cellar/valgrind/3.8.1/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==1087== by 0x100001D9C: __gnu_cxx::new_allocator<char>::deallocate(char*, unsigned long) (in ./putenv_check)
==1087== by 0x100001D2D: std::_Vector_base<char, std::allocator<char> >::_M_deallocate(char*, unsigned long) (in ./putenv_check)
==1087== by 0x1000022E9: void std::vector<char, std::allocator<char> >::_M_assign_aux<__gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > > >(__gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > >, __gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > >, std::forward_iterator_tag) (in ./putenv_check)
==1087== by 0x1000021C4: void std::vector<char, std::allocator<char> >::_M_assign_dispatch<__gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > > >(__gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > >, __gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > >, std::__false_type) (in ./putenv_check)
==1087== by 0x1000020A4: void std::vector<char, std::allocator<char> >::assign<__gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > > >(__gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > >, __gnu_cxx::__normal_iterator<char*, std::vector<char, std::allocator<char> > >) (in ./putenv_check)
==1087== by 0x1000015BF: EnvironmentVariablesManager::put(std::string const&, std::string const&) (in ./putenv_check)
==1087== by 0x10000129E: main (in ./putenv_check)
Now it is more or less clear what is going on. But let’s consider a much more simple example first, purely in C:
#include <unistd.h> #include <stdlib.h> int main() { char *v1, *v2; v1 = malloc(10); strcpy(v1, "x=123"); putenv(v1); free(v1); v2 = malloc(10); strcpy(v2, "x=123"); putenv(v2); free(v2); return 0; }
valgrind also reports about this code (this trace is from Linux):
==523== Memcheck, a memory error detector
==523== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==523== Using Valgrind-3.6.0 and LibVEX; rerun with -h for copyright info
==523== Command: ./putenv_test
==523==
==523== Invalid read of size 1
==523== at 0x4A07CF9: __GI_strncmp (mc_replace_strmem.c:400)
==523== by 0x3E1C235649: __add_to_environ (in /lib64/libc-2.12.so)
==523== by 0x3E1C2353CD: putenv (in /lib64/libc-2.12.so)
==523== by 0x4A0952D: putenv (mc_replace_strmem.c:1165)
==523== by 0x400607: main (in /storage2/home3/ademin/test/env/t)
==523== Address 0x4c28040 is 0 bytes inside a block of size 10 free'd
==523== at 0x4A0595D: free (vg_replace_malloc.c:366)
==523== by 0x4005D7: main (in /storage2/home3/ademin/test/env/t)
==523==
==523== Invalid read of size 1
==523== at 0x4A07D14: __GI_strncmp (mc_replace_strmem.c:400)
==523== by 0x3E1C235649: __add_to_environ (in /lib64/libc-2.12.so)
==523== by 0x3E1C2353CD: putenv (in /lib64/libc-2.12.so)
==523== by 0x4A0952D: putenv (mc_replace_strmem.c:1165)
==523== by 0x400607: main (in /storage2/home3/ademin/test/env/t)
==523== Address 0x4c28040 is 0 bytes inside a block of size 10 free'd
==523== at 0x4A0595D: free (vg_replace_malloc.c:366)
==523== by 0x4005D7: main (in /storage2/home3/ademin/test/env/t)
==523==
==523== Invalid read of size 1
==523== at 0x3E1C235652: __add_to_environ (in /lib64/libc-2.12.so)
==523== by 0x3E1C2353CD: putenv (in /lib64/libc-2.12.so)
==523== by 0x4A0952D: putenv (mc_replace_strmem.c:1165)
==523== by 0x400607: main (in /storage2/home3/ademin/test/env/t)
==523== Address 0x4c28041 is 1 bytes inside a block of size 10 free'd
==523== at 0x4A0595D: free (vg_replace_malloc.c:366)
==523== by 0x4005D7: main (in /storage2/home3/ademin/test/env/t)
==523==
==523==
==523== HEAP SUMMARY:
==523== in use at exit: 0 bytes in 0 blocks
==523== total heap usage: 3 allocs, 3 frees, 492 bytes allocated
==523==
==523== All heap blocks were freed -- no leaks are possible
==523==
==523== For counts of detected and suppressed errors, rerun with: -v
==523== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 6 from 6)
But if to remove the first free(v1);
or move it further down after the
second putenv
, the valgrind report becomes clean, nothing bad happens.
The putenv
function requires that at the moment of setting the new value of
a variable the previous value must still exist and be in valid memory
(literally it cannot be freed or moved to another location).
For some reason putenv
tries reading the previous value when setting the new
one.
Now go back to C++. If the original code:
if (!inserted.second) inserted.first->second = pair;
is replaced to:
if (!inserted.second) inserted.first->second.swap(pair);
the issue disappears (valgrind doesn’t find anything suspecious anymore).
Why? The first code destroys (frees, reallocates) the existing value when
copying the value of pair
into the map. That is why the subsequent call to
putenv
tries reading a freed memory location.
The second code doing swap
only moves the ownership of the data controlled
by pair
to the element of the map, and in turn the data owned by the
element of the map (the existing value) is moved into the pair
variable.
The pair
variable goes out of scope and frees its data only at the end
of the function, clearly after calling putenv
, so by postponing freeing
memory of existing value we allow putenv
to read it without any problems.
This is it! End of story.
Frankly, this is the most complicated bug in C++ memory management I came across recently.
Be aware.
snprintf()
is considered to be “good” for formatting in C in terms of
the buffer overrun problem. But similarly to other functions supporting the
maximum buffer size there is a tricky moment – dealing with the trailing
zero character if the buffer isn’t big enough.
I wanted some clarity in this question, so I wrote the following test program:
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#ifdef WINDOWS
#define snprintf _snprintf
#endif
void test(const int capacity) {
char buf[1024];
int n;
strcpy(buf, "abcdefghijk");
n = snprintf(buf, capacity, "%d", 123);
printf("capacity=%d, n=%d, buf=[%s] (length %d)\n",
capacity, n, buf, (int)strlen(buf));
}
int main() {
test(0);
test(1);
test(2);
test(3);
test(4);
test(5);
return 0;
}
The program tests how snprintf()
deals with the buffer when there is no
room for the full result, and whether it adds the trailing \0
in this case.
I’ll be testing on different systems and compilers.
capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)
capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)
capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)
capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)
capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)
capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)
capacity=0, n=0, buf=[abcdefghijk] (length 11)
capacity=1, n=-1, buf=[] (length 0)
capacity=2, n=-1, buf=[1] (length 1)
capacity=3, n=-1, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)
capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)
capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)
capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)
capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)
capacity=0, n=-1, buf=[abcdefghijk] (length 11)
capacity=1, n=-1, buf=[1bcdefghijk] (length 11)
capacity=2, n=-1, buf=[12cdefghijk] (length 11)
capacity=3, n=3, buf=[123defghijk] (length 11)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)
capacity=0, n=-1, buf=[abcdefghijk] (length 11)
capacity=1, n=-1, buf=[1bcdefghijk] (length 11)
capacity=2, n=-1, buf=[12cdefghijk] (length 11)
capacity=3, n=3, buf=[123defghijk] (length 11)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)
capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)
capacity=0, n=3, buf=[abcdefghijk] (length 11)
capacity=1, n=3, buf=[] (length 0)
capacity=2, n=3, buf=[1] (length 1)
capacity=3, n=3, buf=[12] (length 2)
capacity=4, n=3, buf=[123] (length 3)
capacity=5, n=3, buf=[123] (length 3)
On all UNIX systems (SunOS, Linux, AIX, OSX), except HP-UX, the buffer is not
touched if it is capacity is 0. Also, the trailing \0
is counted in the
length of the output (if the buffer size is 1 it can fit only one character,
and this character is the trailing zero). Finally, the function returns the
length of data which would be written if the buffer has enough room. This
length can be used to allocate another, big enough buffer and call the function
once again.
Alas, on HP-UX, if the buffer cannot fit the result, the function returns -1. In this case it is not clear how to figure out the required length of the buffer. By the dichotomy method?
On Windows the situation is even worse. First, the function returns -1 similar
to HP-UX if the buffer is not big enough. Second, it does not count the trailing
zero in the length of its output, which means it does not append the trailing
zero at all in the situation of the small buffer. Though, Microsoft does not
recommend using snprintf()
at all as a non-thread safe function, and
recommends _snprintf_s()
instead.
It is obvious now why there are so many “a portable snprintf” implementations in the internet.
In my particular case I would be happy using asprintf as an easy to use alternative, but this function is not standard, and, for instance, HP-UX does not support it.
The putenv()
function has a quite unpleasant property. It doesn’t take a
copy of its argument using it directly by pointer. Moreover, the argument
is a non-const pointer. So, it is impossible to pass automatic or
temporary objects, and even passing constant strings (which are persistent
by default) we need to cast them to non-const ones, which is not quite right.
All of this encourages such nonsense like malloc
or strdup
.
So, there is a class below called EnvironmentVariablesManager
. This is a
simple wrapper around putenv()
and getenv()
proving persistent storage
for the values passed to putenv()
(in the form of “name=value”) .
The class is designed to work on Linux, AIX, HP-UX, Solaris and Windows.
EnvironmentVariablesManager.h file:
#ifndef ENVIRONMENT_VARIABLE_MANAGER_H #define ENVIRONMENT_VARIABLE_MANAGER_H #include <string> #include <vector> #include <map> class EnvironmentVariablesManager { public: typedef std::vector<char> VariableContainer; typedef std::map<std::string, VariableContainer> Variables; EnvironmentVariablesManager() {} void put(const std::string& name, const std::string& value); std::string get(const std::string& name) const; void del(const std::string& name); static void PutOSVariable(char* value); static std::string GetOSVariable(const char* name); static bool IsOSVariableSet(const char* name); private: VariableContainer PairToContainer(const std::string& name, const std::string& value) const; Variables vars_; // This class is not copiable. EnvironmentVariablesManager(const EnvironmentVariablesManager&); void operator=(const EnvironmentVariablesManager&); }; #endif
EnvironmentVariablesManager.cpp file:
#include "EnvironmentVariablesManager.h" #ifdef WINDOWS #include <windows.h> #else #include <unistd.h> #endif #include <vector> #include <map> #include <string> #include <iterator> #include <algorithm> #include <cassert> void EnvironmentVariablesManager::put(const std::string& name, const std::string& value) { const VariableContainer pair = PairToContainer(name, value); const std::pair<Variables::iterator, bool> inserted = vars_.insert(std::make_pair(name, pair)); if (!inserted.second) inserted.first->second = pair; char* data = &(inserted.first->second[0]); PutOSVariable(data); } std::string EnvironmentVariablesManager::get(const std::string& name) const { return GetOSVariable(name.c_str()); } void EnvironmentVariablesManager::del(const std::string& name) { put(name, ""); } void EnvironmentVariablesManager::PutOSVariable(char* value) { ::putenv(value); } std::string EnvironmentVariablesManager::GetOSVariable(const char* name) { #ifdef WINDOWS size_t sz = 0; assert(getenv_s(&sz, NULL, 0, name) == 0); if (sz == 0) return std::string(); std::vector<char> value(sz + 1); assert(getenv_s(&sz, &value[0], sz, name) == 0); return std::string(&value[0], sz - 1); #else const char* const value = std::getenv(name); return value ? value : ""; #endif } bool EnvironmentVariablesManager::IsOSVariableSet(const char* name) { #ifdef WINDOWS size_t sz = 0; assert(getenv_s(&sz, NULL, 0, name) == 0); return sz > 0; #else const char* value = std::getenv(name); return value != NULL && *value != '\0'; #endif } EnvironmentVariablesManager::VariableContainer EnvironmentVariablesManager::PairToContainer(const std::string& name, const std::string& value) const { VariableContainer pair; std::copy(name.begin(), name.end(), std::back_inserter(pair)); pair.push_back('='); std::copy(value.begin(), value.end(), std::back_inserter(pair)); pair.push_back('\0'); return pair; }
Unit-tests using std::assert
.
EnvironmentVariablesManager_unittest.cpp file:
#include <iostream> #include <string> #include <vector> #include <cstdlib> #include <cstring> #include <cassert> #ifdef WINDOWS #include <windows.h> #else #include <stdlib.h> #include <stdio.h> #include <unistd.h> #endif #include "EnvironmentVariablesManager.h" void Test_EnvironmentVariablesManager_get_put() { EnvironmentVariablesManager env; assert(std::string("") == env.get("_a_unique_variable_")); env.put("_a_unique_variable_", "b"); assert(std::string("b") == env.get("_a_unique_variable_")); env.put("_a_unique_variable_", "abc"); assert(std::string("abc") == env.get("_a_unique_variable_")); env.put("_a_unique_variable_", ""); assert(std::string("") == env.get("_a_unique_variable_")); } namespace { std::string ReadEnvironmentVariableViaShell(const std::string& name) { #ifdef WINDOWS const std::string shell = EnvironmentVariablesManager::GetOSVariable("ComSpec"); assert(!shell.empty()); const std::string cmd = shell + " /c echo %" + name + "%"; FILE* const f = _popen(cmd.c_str(), "rb"); #else const std::string cmd = "echo $" + name; FILE* const f = popen(cmd.c_str(), "r"); #endif assert(f != NULL); std::vector<char> line(1024, 0); size_t read = 0; while (!::feof(f) && read < line.size()) { const size_t sz = ::fread(&line[read], 1, line.size() - read, f); read += sz; } #ifdef WINDOWS ::_pclose(f); #else ::pclose(f); #endif line.resize(read); std::string trimmed(read, '\0'); std::copy(line.begin(), line.end(), trimmed.begin()); return trimmed.substr(0, trimmed.find_last_not_of("\r\n") + 1); } } void Test_EnvironmentVariablesManager_put_is_propagated_to_child_process() { EnvironmentVariablesManager env; #ifdef WINDOWS const std::string empty = "%__unique_%"; #else const std::string empty = ""; #endif assert(empty == ReadEnvironmentVariableViaShell("__unique_")); env.put("__unique_", "b"); assert(std::string("b") == ReadEnvironmentVariableViaShell("__unique_")); env.put("__unique_", ""); assert(empty == ReadEnvironmentVariableViaShell("__unique_")); } void Test_EnvironmentVariablesManager_must_take_a_copy() { EnvironmentVariablesManager env; char var[] = "12345678"; env.put("var", var); assert(env.get("var") == std::string("12345678")); std::strcpy(var, "abc"); assert(env.get("var") == std::string("12345678")); } void Test_EnvironmentVariablesManager_del() { EnvironmentVariablesManager env; env.put("variable_to_delete", "123"); assert(std::string("123") == env.get("variable_to_delete")); env.del("variable_to_delete"); assert(env.get("variable_to_delete").empty() == true); } void Test_EnvironmentVariablesManager_IsOSVariableSet_set_and_unset() { EnvironmentVariablesManager env; env.put("a", "value"); assert(EnvironmentVariablesManager::IsOSVariableSet("a") == true); env.put("a", ""); assert(EnvironmentVariablesManager::IsOSVariableSet("a") == false); } void Test_EnvironmentVariablesManager_GetOSVariable() { const std::string unique_name = "EnvironmentVariablesManager_GetOSVariable"; assert(EnvironmentVariablesManager::GetOSVariable(unique_name.c_str()) .empty()); const std::string unique_name_pair = unique_name + "=12345678"; char var[1024]; unique_name_pair.copy(var, sizeof(var)); var[unique_name_pair.length()] = '\0'; ::putenv(var); assert(EnvironmentVariablesManager::GetOSVariable(unique_name.c_str()) == "12345678"); } void Test_EnvironmentVariablesManager_PutOSVariable() { const std::string unique_name = "EnvironmentVariablesManager_PutOSVariable"; const char* before = ::getenv(unique_name.c_str()); if (before != NULL) assert(std::string(before).empty()); const std::string unique_name_pair = unique_name + "=12345678"; char var[1024]; unique_name_pair.copy(var, sizeof(var)); var[unique_name_pair.length()] = '\0'; EnvironmentVariablesManager::PutOSVariable(var); const char* after = ::getenv(unique_name.c_str()); assert(after != NULL); assert(std::string(after) == "12345678"); } int run_tests(int argc, const char* const argv[]) { Test_EnvironmentVariablesManager_GetOSVariable(); Test_EnvironmentVariablesManager_PutOSVariable(); Test_EnvironmentVariablesManager_get_put(); Test_EnvironmentVariablesManager_put_is_propagated_to_child_process(); Test_EnvironmentVariablesManager_must_take_a_copy(); Test_EnvironmentVariablesManager_del(); Test_EnvironmentVariablesManager_IsOSVariableSet_set_and_unset(); return 0; } int main(int argc, const char* const argv[]) { run_tests(argc, argv); std::cout << "All tests pass." << std::endl; }
Overall, nothing really intricate but quite handy.
My first POS was VeriFone OMNI-395. It was based on Zilog Z180, up to 1M non-volatile memory (for transaction logs, for instance), a Hayes compatible modem (up to 2400BPS), 12 volts RS232 ports for a PIN pad, a serial printer and other devices (alas, not all RS232 ports had all standard line causing inventing a handmade wheel for the flow control.), and an LCD-screen with loadable fonts.
The architecture was pretty interesting. The user code was executed inside a VM running on top of native Z180 code. It allowed dealing with larger amounts of memory which Z180 can address directly, and implementing dynamically loadable user modules (R-modules). Unfortunately it affected overall performance, and, for example, even table-driven CRC16 on 5-10KB of data took a few seconds. Also, the stack size in the C language was quite limited, and sometimes it was worth implementing, for example, the “sprintf” function manually to avoid random crashes due to stack overflows.
But, frankly speaking, comparing to terminal from other vendors back those days (Injenico, Nurit, etc.) where you had to deal with using raw memory blocks, switching pages manually, etc., VeriFone (TXO) provided almost standard C library. In TXO C you dealt with persistent files, serial ports, LCD and other peripherals with read/write/ioctl functions.
The TXO C compiler supported support also loadable modules (R-modules). Such modules can be loaded on the fly from the user code (similar to overlays or CHAIN in the classic BASIC).
A few days ago The Frost Father presented me an old good OMNI-395. Do you want to see it inside? Indeed!
Turn it on, and, (surprise!) it still runs my firmware.
Inside.
The top board, under the keypad, and the bottom one, with connectors.
If I find the compiler and loader I’ll try to write a little demo.
Alas, there is no official documentation about OMNI-395 hardware, but the programmer manuals only cover the standard library and the VM a little bit.
Of course, OMNI-395 can be used as a generic controller. It can communicate via the standard RS232, plus DTR/RTS can be digital output and CTS/DSR – inputs.
Ideally, it is possible to disassemble the ROM (only 64K of Z180 code), and then reverse engineer the architecture. It will allow programming OMNI-395 directly in Z180 assembly.
Despite of changing the hardware platform a few times after OMNI-395, VeriFone also provides very good backward compatibility in the programmer’s API. So, porting to newer models is quite simple. After OMNI-395 I ported our POS software to OMNI-3350, 3750, VX510, VX610.
An finally, by the #cardpayments tag at Twitter I post pictures of POS, PIN pads, ATM and other plastic card payment machines I come across. Please, join.
The maximum window size is 100x50 with 64 colours. It supports up to 9
windows simultaneously. Each window is controlled independently via
special escape (^[
) sequences.
In general – very useful for small projects on microcontrollers if you don’t bother doing VGA output by yourself (RS-232 output can be implemented even manually if your microcontroller doesn’t support it for some reason). I also tested Serial VGA with Raspberry Pi. The only recommendation is to support the CTS signal line on baud rates faster than 9600. Otherwise the device can loose some characters.
At the moment I have 20 processors (from Intel, AMD, National Semiconductor, NEC, Samsung, Texas Instruments, and also manufactured in the Soviet Union and Czechoslovakia). Amongst chips having a year on the label the earliest is dated back to 1974 and the latest is 1980. All CPUs except one are fully functional. I tested them on my Радио-86РК.
Testing revealed that all processors are identical according to the CPU Exerciser except clones from AMD. The AMD processors, AM8080 and AM9080A, behave differently performing the bitwise AND operation (ANA and ANI instruction). The original Intel CPUs and non-AMD clones set the AC (half-carry) flag to the value of the 3rd bit (A3) from the bitwise OR between the accumulator and the argument of ANA or ANI. The AMD clones always zero the AC flag in the ANA and ANI instructions. I don’t know why the original Intel CPU calculates the AC flag in such a weird way.
The double click on the pictures flips the top and the bottom views.
Interestingly, AMD i8080-compatible chips were reverse-engineered from schematics literally stolen from Intel. So, the Intel vs AMD war began from 8080.
This chip is faulty.
There is the original mini-datasheet in Russian: page 1 and page 2.
This is how I took pictures of all these chips in a quite technological way using one iPhone and two Raspberry Pi:
I hope the collection will grow. I still have only one chip in white ceramic packaging (KR580VM80). If you know i8080 clones from other manufacturers, which are not listed here, I’d appreciate if you let me know.
If you would like to donate any i8080 chip I’ll be happy putting a reference to you next to the image of the chip.
And you know what, after joining the “dark side” army of Apple users, particularly with Macbook Air, I become much more tolerant paying for software. And the main reason is, you guess?, the easiness of a purchase. With AppStore you just “click-click”, and an application jumps into your Applications in a few seconds. Purchasing directly on web sites is usually simple and smooth as well, especially if you pay using PayPal or Google Wallet.
Moving even further, I listed some apps which I am happy to pay for.
Now a couple of apps which I also pay for but only because there are no adequate alternatives. I’m more or less happy with functionality but I don’t like their pricing (in short – too much in my view).
However, there is an app, which I’m not willing to pay. Its price is ridiculous. This is the Microsoft Office. Fortunately, the company I work for provided me a license for this product, so I can work with working documents on Mac. When it gets stopped I’ll switch to the Apple alternatives (Pages, Numbers and Keynotes).
And finally, there is an app called Radare. It is free and opensource, and I donated some amount to support this great project (I hope not last time).
Of course, there are tons of other free apps (iTerm, 7z, muCommander, Skype, VirtualBox, GitHub, DOSBox, KeePassX, etc.), which I also use but this is a different story.
Agreed, the game is trivial, but I just liked it. Also, I tried ascii.io recording an asciicast, so you can evaluate gameplay - http://ascii.io/a/1715.
The source is available at GitHub’е, but you can simply grab it below.
// Originally taken from http://itblogs.org/c-konsolnaya-zmejka/. #include <iostream> #include <cstdio> #include <cstdlib> #include <ctime> #ifdef WINDOWS #include <windows.h> #include <conio.h> #else #include <unistd.h> #include <termios.h> #include <sys/select.h> #define STDIN_FILENO 0 #define NB_DISABLE 0 #define NB_ENABLE 1 #define Sleep(x) usleep(x*1000) int kbhit() { struct timeval tv; fd_set fds; tv.tv_sec = 0; tv.tv_usec = 0; FD_ZERO(&fds); FD_SET(STDIN_FILENO, &fds); select(STDIN_FILENO+1, &fds, NULL, NULL, &tv); return FD_ISSET(STDIN_FILENO, &fds); } void nonblock(int state) { struct termios ttystate; // Get the terminal state. tcgetattr(STDIN_FILENO, &ttystate); if (state == NB_ENABLE) { // Turn off canonical mode. ttystate.c_lflag &= ~ICANON; // Minimum of number input read. ttystate.c_cc[VMIN] = 1; } else if (state == NB_DISABLE) { // Turn on canonical mode. ttystate.c_lflag |= ICANON; } // Set the terminal attributes. tcsetattr(STDIN_FILENO, TCSANOW, &ttystate); } int getch() { return fgetc(stdin); } #endif int snake_size, change_x, change_y, coordinates_x[1000], coordinates_y[1000]; int food_x = -1, food_y = -1; char symbol, a[1000][1000]; const int N = 13, M = 17, INTERVAL = 200; void change_direction() { symbol = getch(); switch (symbol) { case 'w': if (change_x != 1 || change_y != 0) { change_x = -1; change_y = 0; } break; case 'a': if (change_x != 0 || change_y != 1) { change_x = 0; change_y = -1; } break; case 's': if (change_x != -1 || change_y != 0) { change_x = 1; change_y = 0; } break; case 'd': if (change_x != 0 || change_y != -1) { change_x = 0; change_y = 1; } break; #ifndef WINDOWS case 'q': nonblock(NB_DISABLE); std::exit(0); #endif default: break; } } void show_table() { #ifdef WINDOWS system("cls"); #else system("clear"); #endif for (int i = 0; i <= N + 1; ++i) for (int j = 0; j <= M + 1; ++j) std::cout << (i == 0 || j == 0 || i == N + 1 || j == M + 1 ? '#' : a[i][j]) << (j <= M ? "" : "\n"); } void clear_snake_on_table() { for (int i = 1; i <= snake_size; ++i) a[coordinates_x[i]][coordinates_y[i]] = ' '; } void show_snake_on_table() { if (change_x == 1 && change_y == 0) a[coordinates_x[1]][coordinates_y[1]] = 'v'; if (change_x == -1 && change_y == 0) a[coordinates_x[1]][coordinates_y[1]] = '^'; if (change_x == 0 && change_y == 1) a[coordinates_x[1]][coordinates_y[1]] = '>'; if (change_x == 0 && change_y == -1) a[coordinates_x[1]][coordinates_y[1]] = '<'; for (int i = 2; i <= snake_size; ++i) a[coordinates_x[i]][coordinates_y[i]] = '@'; } bool game_over() { for (int i = 2; i <= snake_size; ++i) if (coordinates_x[1] == coordinates_x[i] && coordinates_y[1] == coordinates_y[i]) return true; return false; } void check_coordinates() { if (coordinates_x[1] > N) coordinates_x[1] = 1; if (coordinates_x[1] < 1) coordinates_x[1] = N; if (coordinates_y[1] > M) coordinates_y[1] = 1; if (coordinates_y[1] < 1) coordinates_y[1] = M; } void next_step() { clear_snake_on_table(); for (int i = snake_size; i >= 2; --i) { coordinates_x[i] = coordinates_x[i - 1]; coordinates_y[i] = coordinates_y[i - 1]; } coordinates_x[1] += change_x; coordinates_y[1] += change_y; check_coordinates(); if (coordinates_x[1] == food_x && coordinates_y[1] == food_y) { snake_size++; food_x = -1; food_y = -1; } show_snake_on_table(); if (game_over()) { std::cout << "You're looser! \n"; #ifdef WINDOWS system("pause"); #endif std::exit(0); } } bool food_check() { if (food_x == -1 && food_y == -1) return false; return true; } void place_food() { std::srand(std::time(NULL)); for (int i = 1; i <= 9; ++i) { int x = std::rand(), y = std::rand(); if (x < 0) x *= -1; if (y < 0) y *= -1; x %= (N + 1); y %= (M + 1); if (x == 0) ++x; if (y == 0) ++y; if (a[x][y] != '@' && a[x][y] != 'v' && a[x][y] != '^' && a[x][y] != '<' && a[x][y] != '>') { food_x = x; food_y = y; a[x][y] = '+'; return; } } } void standart_settings() { snake_size = 2; coordinates_x[1] = 1; coordinates_y[1] = 2; coordinates_x[2] = 1; coordinates_y[2] = 1; change_x = 0; change_y = 1; } int main() { standart_settings(); #ifndef WINDOWS std::memset(a, ' ', sizeof(a)); nonblock(NB_ENABLE); #endif while (true) { if (kbhit() != 0) change_direction(); next_step(); if (!food_check()) place_food(); show_table(); Sleep(INTERVAL); } }
By the way, if you have similar cool console stuff, please share.
Recently Geoff has released a new, updated colour version modification having a few extra features along with the colour (new and improved features are marked with the asterisk):
MMBasic allows to take full advantage of all these features. It is even possible implementing the PIC32 timer interrupt routines directly in BASIC.
As previously I have ordered the new Maximite Kit from Altronics.
There are no on-surface planar elements which makes the life of soldering laymans like me much easier. There is only one planar capacitor (C10) for some reason. It is quite small, so proper soldering it took me some time to avoid shortening on the board.
All done.
The Original and Colour Maximite.
In general, the kit from Altronics is very high quality. Recommended.
This is what MMBasic allows a user to do with colours.
Well, no colours though, but just classics.
As the author explains, generating colours requires the 100-pin version of PIC32 with three SPI channels. Obviously, it also requires three times much work processing the colour information. Because the crystal still works on the same frequency as previously, 80MHz, the colour Maximite works a bit slower than its black-and-white predecessor. But along with the firmware providing colours, there is a black-and-white version of MMBasic for the new Maximite having all new features except the colour video, so it works with the same speed as the original Maximite.
Well, as previously I have a lot of fun using Maximite, and strongly recommend it if you are up building a nice powerful microcomputer in a few hours.
On Windows Splunk can take information from Windows Events, Performance Counters or Registry.
For Splunk logs are textual files split into lines. When indexing logs Splunk parses log lines into fields, for example, in the “name=value” format (which is also configurable though). Then using a special query language called SPL it allows manipulating with these fields: sorting, aggregation, computed fields, creating views and tables, external dictionary lookups (for instance, to RDBMS), and, of course, creating various graphic representations of data. Even though normally SPL operates with individual log lines, it supports multi-line queries, which can be useful, for example, to extract transactions or any other activities generating multiple log records.
As Splunk creators state, all logs kept and indexed by Splunk despite of their age are equally available for querying. There is no concept of archiving. Of course, machines running Splunk should correspond to the volume of logs being processed.
Also Splunk states itself as “Google for logs”, but let’s leave it without comments.
The UI of Splunk is web. It allows creating dashboards, which in turn can be presented as a Splunk app. Splunk has its own app store, where most of apps are free though. There are apps already developed, for example, for UNIX syslog, Apache logs, Microsoft Exchange, etc.
Splunk software is freely available on the official website. The licensing model is based on volumes of logs processed daily. Under some small threshold Splunk is free. This threshold is surely enough for evaluation.
I would recommend a book called Exploring Splunk for those who want quickly dig into Splunk and figure out its main features. This book provides very solid overview of SPL and other concepts how Splunk works.
In this article I would like to show a real example of using Splunk. This exercise can be repeated in half an hour. The only prerequisite is to download and install Splunk on your operating system. Then just follow my instructions.
The example is a bit unusual though. Traditionally, logs are used for post factum analysis. But nothing really prevents from picking up updates on the fly and building “live” indicators based on logs. Agreed, my example may look a bit artificial, but I want to show how quickly to feed data into Splunk and formalize it, and finally create a dynamic UI with it.
Let’s begin with a simple script in Ruby. It generates log records containing a number representing its completion (from 0 to 100%).
require 'date' duration = 60*1 update_period = 0.5 i = 0 while i <= duration do progress = i * 100.0 / duration msg = "%s progress=%05.2f\n" % [DateTime.now, progress] puts msg open("logs/my.log", 'a') { |f| f << msg } i = i + update_period sleep update_period end
The log records may look like:
2012-11-23T15:58:54+00:00 progress=45.00
2012-11-23T15:58:55+00:00 progress=45.83
2012-11-23T15:58:55+00:00 progress=46.67
2012-11-23T15:58:56+00:00 progress=47.50
2012-11-23T15:58:56+00:00 progress=48.33
2012-11-23T15:58:57+00:00 progress=49.17
2012-11-23T15:58:57+00:00 progress=50.00
We will develop a dashboard in Splunk, which displays the percentage in the form of a cool indicator taking updates from logs.
For simplicity we run everything on a single machine, and Splunk will be taking log updates directly from a file.
So, you have already installed Splunk, and if you open “http://localhost:8080” in your browser you should be able to login as the “admin” user.
Then in menu follow to: “Manager -> Data Inputs -> Add data -> A file or directory or files”. Here we specify the file and directory name with the logs (in our case just a single file).
Confirm creating a source of logs (source type). As I said, logs can be delivered to Splunk in various ways. Each way can be named and addressed as a source type.
The log has been added. We see that Splunk has picked up the file and already parsed its lines into fields. Out of the box Splunk understands many date and time formats, but it is configurable.
Now we give a name to our source type - “test_logging” and save the settings.
Now we get back to the main page and type in our first SPL query in the search bar (in red):
sourcetype="test_logging" | table progress as float
It says: take logs from the “test_logging” data source, create a table with one column called “progress” taking values from the corresponding field, set the type of “progress” to float. At the bottom (in blue) we see the result of the query (own log already has some data). SPL uses the concept of UNIX pipes (|) feeding output from one command to the next one.
So, we have the table. Now let’s create a widget. Due to simplicity of our data (one single field) we can represent it as a speedometer gauge. Click on “Formatting options” (in blue) and set the chart type to “radial gauge” (in red). And here is our cool widget:
The first widget is ready. Now let’s create an alternative one looking differently. It will also represent the value of the progress but in the form of a horizontal bar moving from left to right. This is the query:
sourcetype="test_logging" | table _time progress | head 1
It says: using data from the “test_logging” source type create a table with two fields, “_time” and “progress”, and then take only the first line from it. Default sorting is descending by “_time”. At the bottom (in green) we see the result of the query.
Now click on “Formatting options”, choose “bar” (in green), set the interval for the Y-axis from 0 to 100. For some reason, the X-axis is vertical here (this axis will represent “_time”), and the Y-axis is horizontal (this one will show “progress”). Because our query on the previous picture already shown “100”, the widget is fully filled.
Describing the first widget I deliberately skipped how to save the query. We will do it now. We click on “Create” and then “Dashboard panel…” (in red) and save the settings. We name the first widget as “Speedometer” and the second - “Progress bar”.
Along with saving the first widget we will be asked to create a new panel, a dashboard. Let’s call it “Test logging”. When saving the second widget we will add it to the already created one.
Finally, we click on “Dashboards & View”, select your dashboard and it will be shown like:
There is no data here, so the panels are empty. We see the name of the dashboard (in red), the names of the widgets (in yellow) and two “Edit” buttons (in blue) allowing updating queries and visual representation on the fly. Before starting our Ruby script, we need to click on the both “Edit” buttons and set the time interval from “rt-1s” to “rt”. Such interval means getting updates each second.
Now let’s click on “On” and kick off the script!
A little video how it works:
That’s it.
The example, as I said, is very simple, but I hope it gives you as least initial understanding of Splunk.
Main distinctive features of Propeller:
The Spin language was designed to make concurrent and multi-core programming easier. This language is a mixture of procedural and object-oriented approaches.
Below is an example of code in Spin launching a function to spin on three cogs. The code is really easy to read and understand.
CON
_clkmode = xtal1 + pll16x 'Establish speed
_xinfreq = 5_000_000 '80Mhz
OBJ
led: "E555_LEDEngine.spin" 'Include LED methods object
VAR
byte Counter 'Establish Counter Variable
long stack[90] 'Establish working space
PUB Main
cognew(Twinkle(16,clkfreq/50), @stack[0]) 'start Twinkle cog 1
cognew(Twinkle(19,clkfreq/150), @stack[30]) 'start Twinkle cog 2
cognew(Twinkle(22,clkfreq/100), @stack[60]) 'start Twinkle cog 3
PUB Twinkle(PIN,RATE) 'Method declaration
repeat 'Initiate a master loop
repeat Counter from 0 to 100 'Repeat loop Counter
led.LEDBrightness(Counter, PIN) 'Adjust LED brightness
waitcnt(RATE + cnt) 'Wait a moment
repeat Counter from 100 to 0 'Repeat loop Counter
led.LEDBrightness(Counter,PIN) 'Adjust LED brightness
waitcnt(RATE + cnt) 'Wait a moment
The cognew
function kicks off the function Twinkle
on three cores
and parameterizes each one with a frequency and stack.
The diagram of the Propeller architecture:
The name “Propeller” comes from how the Hub controlling resource sharing switches between cogs. It does round-robin similar to the spinning propeller.
I don’t want to go deeper into Propeller in this post because this is a long story. Instead I put a list of very useful books at the end. They provide the exhaustive details.
But I’d like to share about an interesting project called “The Pocket Mini Computer” based on Propeller (P8X32A). This project uses the “P8X32A QuickStart” evaluation board as a base.
The photo from the official website:
In fact, the author offers the evaluation board, plus the extension, which has VGA, microSD, PS/2, sound and Wii Gameport interfaces. Optionally, the 32KB SRAM chip can be added.
The point of the project is a BASIC interpreter running on this computer. It converts it to a micro-computer a-la from 80s. The interpreter is written in Spin (sources are open). The dialect is quite limited though: no arrays, real and string variables, one letter variable names etc. Nevertheless, BASIC provides access to all peripherals including the SD-card. Plus, it allows running native binaries which can be developed in assembly, C (Parallax has a GCC version for Propeller) or Spin.
Below there are a few pictures of the kit to understand how it looks like. As I said, PMC is based on the P8X32A evaluation board, so only the extension board requires assembly (soldering).
Almost everything is in place.
The sandwich is ready.
A small demo of graphics capabilities.
It is not easy to call it “a general purpose microcontroller”. In my subjective view it fits perfectly into certain applications, like, for instance, games consoles or mini-vending machines because it can generate a good quality video signal out of the box. But, on the other hand there are no built-in PWM, ADC/DAC, flash memory, triggers, interrupts, and the creators recommend implementing such functionality manually or using external chips. In the books listed in the end there are plenty examples of such applications.
Conslusion: the very interesting and worth considering architecture.
This is a very interesting and entertaining project. Of course, it is obvious that BASIC is very limited in resources, for instance, RAM. Maximite, for example, based on PIC32, is much more powerful. It can easily run not only its powerful MMBasic, but, for example, RetroBSD or even emulate Radio-86RK.
But, $39 is the great offer for those who want to play with Propeller having complete hardware.
Here is a list of books I read (regarding the architecture) and skimmed (regarding projects). All are recommended.
A short, concise and easy book for beginners. It describes many projects on Propeller. One of the authors is the PMC inventor.
Getting Started With the Propeller
A very solid and easy to read book. It explains in all details the architecture of Propeller. Though the author doesn’t cover programming in assembly, only in Spin, but he precisely shows how to use properly and benefit from the original multi-core architecture of Propeller. The first section covers the internals of Propeller. Following sections contain various projects (can be skipped on the first reading).
Programming the Propeller with Spin : A Beginner’s Guide to Parallel Processing (Tab Electronics)
A collections of real applications with Propeller from its creators.
Programming and Customizing the Multicore Propeller Microcontroller : The Official Guide
A small lyrical digression.
Does any know that Peter Norton didn’t write the famous Norton Commander? The Norton Commander had been invented and then maintained by John Socha up to its “canonical” version 3. Socha used quite an unusual approach in those days mixing the assembly language with C when most of “true” developers programmed purely in assembly. It allowed developing sophisticated but still fast and efficient applications much quicker.
Okay, go back to the book. It liked the idea of buying old books almost for nothing just for the sake of holding them in hands.
Wonderful. They sent me a decommissioned library copy carefully glued by the sticky tape.
I personally don’t like when people write in books or fold page corners, but this looks very touching – somebody did drilled this stuff.
Is it still possible NOT to understand how EQU works?
Or DUP?
Or stack?
Or how to intercept interrupts in DOS?
By the way, it turned that I have got the third edition of the book having a couple of extra chapters about 286 and 386 comparing to its first edition which I read years ago in Russian.
The bottom line – this book was and still is fantastic.
Through the book they explain step by step a process of developing a program called “dskpatch”, a visual editor allowing viewing and modifying individual sectors on floppy or hard drives. In the end it turns into an advanced product split into modules and having dozens of generic routines easily re-usable in any program. Starting from the very beginning you have to learn the basics of 8086, then BIOS and DOS functions, the direct access to video memory and keyboard.
But the main thing is that this book explains how to write really big projects in assembly. For example, my real opening was that if each subroutine always preserves the registers which are not to pass parameters and to return the result the amount of “unpredictable side effects” drops down substantially. Of course, today it sounds like information from a nursery level, but even in the nursery somebody has communicate it.
In the appendix there is the full carefully commented listing of “dskptach”.
Also, the third edition has a few chapters about mixing the assembler and high level languages like C and PASCAL. Passing function parameters to and from the assembler routines is explained in all details. As a bonus, there is a library with an unambiguous name – SOCHALIB congaing a lot of very useful routines for the screen, mouse and the keyboard. It was delightfully impressed that a routine putting a character on the screen used a pre-calculated lookup table to compute an address of the screen line base instead of multiplying Y times 80 directly. I suspect the code of the library resides somewhere inside the Norton Commander and Norton Utilities, and it explains why these products worked blazingly fast.
The conclusion: strongly recommended as an example of the perfect textbook.
How cool is that? Let’s check it out. It has arrived:
The parts:
Let’s begin its “by slice” composing:
After the first two slices let’s put the RPi (remember that it’s design doesn’t have even holes for screws):
Slice by slice:
Now screws and nuts…
The device is ready!
Frankly I just love Raspberry Pi. It is difficult to imagine a better device for schools and universities for studying computers. Only Maximite could compete if we need a bit more lower level, closer to hardware.
Before my RPi’s “case” from SK Pang looked like this:
Incredibly useful device if you need access to GPIO, but as a “normal” case Pibow is unbeatable.
BEGIN
if (Hi(Code) == 0)
BEGIN
BAsmCode[0] = Lo(Code); CodeLen = 1;
return True;
END
else if (MomCPU <= CPU1802)
BEGIN
WrError(1500);
return False;
END
else
BEGIN
BAsmCode[0] = Hi(Code); BAsmCode[1] = Lo(Code); CodeLen = 2;
return True;
END
END
(grabbed from code1802.c).
Frankly, it is not easy to recognize the language. Of course, this is C but initially it was Pascal, and then the author had converted in to C trying to minimize the amount of changes.
The project is called “Macro-assembler AS”, by Alfred Arnold. I use it as an Intel 8080 macro-assembler available on Mac, Linux and Windows. Frankly I haven’t seen anything better so far.
As the author explains, the project was started in Turbo Pascal, but after Borland had abandoned the DOS version but free implementation of the modern Pascal didn’t exist yet, he decided to convert the sources into C and continue development in C. Despite of overall doubtfulness of such approach, to me, the author had managed it very well. The project hadn’t die as it usually happens with big projects after rewrite. But though I have cloned it for myself and started using successfully, I doubt that I want to contribute because it requires coding in this weird “Pascal on C steroids” dialect. Alas, this is inevitable due to that conversion, otherwise the project may turn into mess. If to surf amongst the sources of AS it is easy to find a lot of tricks like “how to make C to be Pascal”.
Nevertheless, all the best to Alfred with his great project. Again, the project moves forward and becomes better.
By the way, Alfred has a very cool collection of old hardware which makes me feel envy.
Introduction to 8080-8085 Assembly Language Programming (Self-teaching Guides)
A reasonable question “why”? Especially considering that in the last couple of months I have re-written from scratch two emulators of the Intel 8080 microprocessor, in C and JavaScript. Do I still have any questions about 8080 assembly? No.
Frankly, I just heard about this book as a perfect example of the assembly textbook. Something similar to the famous Maurer’s book on programming on 6502 assembly for Apple computers. I believe the digitized copies are available because the book was published years ago back in 1981, but I had come across a very cheap hardcopy (~6 pounds including delivery) at Amazon, second-hand, of course.
Have no idea why but when holding such old books in hands I feel some unexplainable excitement from all these hand-made annotations, old-fashion fonts for code and punch card forms.
And in fact the book is great. If, just in some weird and unusual case you decide to learn the 8080 assembly language, but you have never heard about 8080 and the assembly language in general, this book is the perfect option.
It begins from the very beginning – dealing with binary, decimal and hexadecimal calculations; then it follows through groups of instructions with many examples – array indexing, multiplication and division, BCD representation, the concept of stack and so on. Each chapter is concluded with exercises.
The very good book. Recommended.
git diff --shortstat rel-2010 rel-2011
It prints something like this:
12 files changed, 2462 insertions(+), 488 deletions(-)
If your project isn’t in git, you can quickly import the branches you want to analyse in a temporary git repo.
I know that C is THE low-level assembler, and when it is used properly it generates code very close to the assembly. But nevertheless there are some systems still where the C compiler is difficult to implement efficiently. For example, I wanted a C compiler for Intel 8080 to write a program for Radio-86RK. So far, I have found only a couple derivatives of the famous Small-C – smallc-85 and smallc-scc3 which compile and work.
Unfortunately, even for trivial code:
main() {
static char a;
for (a = 1; a < 10; ++a) {
++a;
}
}
It generates the following rubbish:
;main() {
main:
; static char a;
dseg
?2: ds 1
cseg
; for (a = 1; a < 10; ++a) {
lxi h,?2
push h
lxi h,1
pop d
call ?pchar
?3:
lxi h,?2
call ?gchar
push h
lxi h,10
pop d
call ?lt
mov a,h
ora l
jnz ?5
jmp ?6
?4:
lxi h,?2
push h
call ?gchar
inx h
pop d
call ?pchar
jmp ?3
?5:
; ++a;
lxi h,?2
push h
call ?gchar
inx h
pop d
call ?pchar
; }
jmp ?4
?6:
;}
?1:
ret
Agreed, there are a lot of questions to the compiler, but in general Intel 8080 isn’t quite friendly for the C compiler: no multiplication and division, no indirect addressing on the stack, etc.
Okay, back to Forth. When thinking about Forth for I8080 I wrote a handy macro assembler (it’ll be a separate blog post) and along the way remembered about my old project back to FidoNet times: F-CODE. To obfuscate the code against tracing and disassembling I implemented a micro Forth kernel with direct threading.
“Implemented a micro Forth kernel” sounds “cool” but in reality the direct threading code interpreter is almost trivial:
; F-Code Address Interpreter
GetNext$: cld
mov si, IP$
lodsw
mov IP$, si
retn
CALLR$: add RP$, 2
mov bp, RP$
mov ax, IP$
mov [bp], ax
pop word ptr IP$
next
RETR$: mov bp, RP$
mov ax, [bp]
mov IP$, ax
sub RP$, 2
next
NEXT$: call GetNext$
jmp ax
osPush$: call GetNext$
push ax
next
NEXT MACRO
jmp NEXT$
ENDM
Plus, a few extra primitives implemented in assembly:
; Adc ( a b -> c isCarry )
; if a+b>FFFF isCarry = FFFF else isCarry=0
osAdc$: pop ax dx ; -> a b
add ax, dx
sbb dx, dx
push ax dx ; c isCarry ->
NEXT
; osSwap ( a b -> b a )
osSwap$: pop ax bx
push ax bx
NEXT
; osRot ( a b c -> b c a )
osRot$: pop ax bx cx
push bx ax cx
NEXT
osPut$: add RP$, 2
mov bp, RP$
pop word ptr [bp]
NEXT
osGet$: mov bp, RP$
push word ptr [bp]
sub RP$, 2
NEXT
osDrop$: add sp, 2
NEXT
; osNor ( a b -> a NOR b )
osNor$: pop ax bx
or ax, bx
not ax
push ax
NEXT
osTrap$: int 3
NEXT
; osPeek ( addr -> value )
osPeek$: pop bx
push word ptr [bx]
NEXT
; osPoke ( Value Addr -> )
osPoke$: pop bx ; -> Value Addr
pop word ptr [bx] ; ->
NEXT
Now we have a complete stack machine which we can program. Obviously, when tracing or disassembling the threaded code there are only jumps back and fourth making it very difficult for reverse engineering (the purpose of the obfuscation). If you’re curious you may try digging into fcode.com. This little program asks a password which you need to guest/crack/patch/etc. Note that this is a DOS binary, so it needs, for instance, DOSBox to run.
For example, code to compute CRC on this stack machine:
CalcCRC: CALLR ; ->
ofPush 0 ; CRC
ofPush 0 ; CRC 0
ofPeekb Buffer+1 ; CRC 0 Size
$For ; CRC
osI ; CRC i
ofPush Buffer+2 ; CRC i Buffer+2
osAdd ; CRC Addr
osPeekb ; CRC Value
osExch ; CRC Value*256
$For 0, 8 ; CRC Value
osShl ; CRC Value*2 isCarry
osRot ; Value*2 isCarry CRC
osSwap ; Value*2 CRC isCarry
osRcl ; Value*2 CRC*2 isCarry
$If <>0 ; Value*2 CRC*2
ofXor 8408h ; Value*2 CRC*2^Const
$Endif
osSwap ; CRC*2 Value*2
$Loop ; CRC Value*2
osDrop ; CRC
$Loop ; CRC
RETR
Awesome, right?
When working on F-CODE a simple preprocessor for the assembler language was born allowing to write code like this:
lea dx, msg2
cmp bh, 3
$if <>0
lea dx, msg1
$else
hlt
$endif
cmp dx, 0C0DEh
$if =0
lea dx, msg2
$endif
mov cx, 2
$Do
$Do
cmp ax, 1
$EndDo =
dec cx
$EndDo Loop
Store ds, si, ax
$Do
cmp al, 1
$if <>0
$ExitDo
$endif
Store ax, bx, cx, es, bp
...
Restore
$ContDo
$EndDo Loop
Restore
Similar to other projects back in DOS times I wrote the preprocessor in Turbo Pascal.
Of course the F-CODE project now has predominantly historical value, although nothing stops to implement the interpreter, for instance, in JavaScript and then re-use existing primitives without any changes.
The F-CODE project is available at GitHub – https://github.com/begoon/fcode.
It requires TASM/TLINK and Turbo Pascal 5.+ to build. Obviously, it also requires DOS to run.
P.S. By the way, people develop quite sophisticated software in Forth. For example, nnbackup is fully written in Forth.
Lighter, thinner, faster.
The new connector, plus the headset is at the bottom now.
The new connector (male).
People are seriously stuck to desks with the iPhone 5.
By they way, the new 3D-maps in iOS6 are fantastic! Unfortunately, not available for my iPhone 4 (not S).
The traditional sound check on the right website.
A little followup.
Regardless that I’m fully on Apple products, I believe that I still have at least a little bit of common sense. If some Apple products are bad or even shit (like Magic Mouse, for instance) I joyfully state that. But brothers! The iPhone 5 is an amazing product of design and engineering. God bless Apple’s designers and engineers. You may hate Apple for their pathos and show off, you may try suing them and even win. But! (Samsung, for example) Please! Create something even close to the iPhone 5. And all Apple’s money and glory will be yours.
Two compression libraries were discovered:
The selection criteria were:
The both libraries compiled and worked for PIC32 without any issues.
On my data miniz provided the 0.78 compression ratio and minilzo - 0.71. So, both didn’t squeeze my data into the 512KB flash of PIC32.
Also miniz uses only the output buffer for decompression, but minilzo requries at least 16K static buffer.
P.S. I also tried XZ Embedded (LZMA2). It looked that it compressed much better but it requires malloc/free API, so I didn’t manage to build it on PIC32 without extra development.
I wanted a simple and quick app helping me to remember a list of drink in the round. Plus, I wanted to write something for iOS in general. Eventually, I ended up with this:
Quick and dirty, but it solves the problem in 100%. I personally use this app all the time. Unfortunately, AppStore has rejected it as the app having too limited functionality. Well, bad for them, good for me. I have learned a lot developing this app.
I had to learn the following classes:
The full project is available at GitHub - https://github.com/begoon/buyround. Please, feel free to checkout and build it.
As I wrote previously about my first iPhone app - US Visa, the quality of the artwork, especially, icons, plays the crucial role in the success of a mobile application. I’ve googled one (below) but it is much better to design your own one.
P.S. For the sake of making this post complete I’ve decided to attach a photo of absolutely inhuman conditions of creating it.
karateka.dsk.gz
file and load it into the AppleWin
emulator.
There is a DOS version, but, unfortunately, the gameplay was slightly changed. In the Apple’s version the opponents inside the house never approach you first. There was a trick: a little step towards and immediately step back. This was the only way to force them to attack you. Then you can low-kick them. In the DOS version they always attack first, so just low-kick.
But for the DOS version there is a magic patch allowing to speedrun the game.
Immortality
KARATEKA.EXE
00003066: 48 90
00003D7E: 48 90
Steel breast
KARATEKA.EXE
0000306E: 83 C6
0000306F: 3E 06
00003072: 3F 01
00003073: 7E 75
Flying kick for humans
KARATEKA.EXE
00002F30: 7E 00
Flying kick for bird
KARATEKA.EXE
00002E2F: 7F 00
00002E30: 01 00
00002E34: 3D 25
00002E35: 04 00
00002E44: 06 00
Kill bird by first kick
KARATEKA.EXE
000031BA: 85 33
Kill humans by first kick
KARATEKA.EXE
00002F3A: 85 33
For example:
Don’t miss at the end when the princess hopelessly kicks you when you approach in the stance.
P.S. Unfortunately, the sources of this spectacular game are still closed.
Its DOS implementation is a quite precise replica perfectly working on modern systems in DOSBox:
dosbox -exit BOLO.COM
Years ago I created a patch to know what happens at the end:
Unlimited lives
BOLO.COM
00000174: 4F 00
00000178: DF 4A
00000179: FE FF
Unlimited fuel
BOLO.COM
00000A38: 06 04
Immortality & pass through the walls
BOLO.COM
00000C61: 01 00
0000172D: 01 00
00001817: 01 00
With this patch you can play like this:
Unfortunately, there are no sources of the original BOLO available, so porting to SDL, for instance, is fully about reverse engineering of the original binaries (either of the Apple II version or the DOS clone), which is not that easy. I know there are some other BOLO clones for Windows are available, but they are physically different games, not original BOLO.
It costs 128€ + 25.80€ VAT for the 8 channel model. They have stock in the UK, so delivery was free, fast and no customs involved.
At the beginning I had a cunning plan to buy, play for a while and then return. They provide an option for full money back within a month without explaining anything. This is very good. Frankly, if there was no such option, I would never buy it.
It turned out to be a match box size thing in the solid metal case.
The complete set.
Connecting…
To begin with I connected it to a serial port on Raspberry Pi, which I was using as a telnet console.
Here is a screen shot of the capture. We see that the ls
command is flying
to one direction and being echoed back from the second one.
Now the GMC-4 keyboard. This is a scanned keyboard where one (out) port is used to set a “flying” pin selecting the current row (or column) of buttons, and the second (in) port represents the state of buttons in the selected row.
I planned to simulate the keyboard, so I needed timings of the scanning signal. I connected all four scanning lines to four channels, and got the following:
Now it is crystal clear what’s going on. It allows to measure a variety of parameters of the signal, scale it and compare against previous runs. There is a good quality video available on the official website showing some use cases in details. My status in electronics is a layman, so I was simply impressed by the ease of use. The guy, Viacheslav, who recommended me this device, says that for debugging a complicated digital circuit design it is crucial to be able to capture much more than one or two channels simultaneously.
As you have probably realised, I didn’t return the device, but I had a complaint though. I wasn’t able to capture on the highest frequency of 24MHz. The Logic software complained about lack of USB bandwidth when transferring data from the Logic to the computer. The Support suggested an issue with my particular USB hub, but I tried different setups and the problem still stands. But they were ready to take the device back and refund anytime. Regardless, I liked the device and kept it. It perfectly solves all my current needs.
Recommended.
comrade@gmail.com
,
e-mails addressed to com.rade@gmail.com
or even to c.o.m.r.a.d.e@gmail.com
will be delivered to the first address.
It’s cool but quite useless. There is another interesting feature. E-mails sent to the addresses below:
comrade+fred@gmail.com
comrade+john@gmail.com
comrade+peter@gmail.com
…comrade+a_somebodies_name@gmail.com
will be also delivered to comrade@gmail.com
. Then it is possible to set
some filtering rules in Gmail redirecting these e-mails to appropriate people
depending on that extra bit in the “From:” address between +
and @
symbols.
This may help organizing a free “pseudo” mail server with Gmail.
By the way, there is no prize for solving this problem except the honour and a link which I can put with the solver’s name.
Please, don’t post the solution anywhere in comments letting others to try solving it independently.
That was the first, simplest level, but how about this one?
The game perfectly works in DOSEmu: DOSEmu -exit pusher.exe
.
Lazy people can checkout a quick video instead:
I was curious how 60 maps fitted into such a tiny executable. After a bit
of fiddling with IDA I wrote a simple program to extract the maps from the
pusher.exe
binary and print them out. For example:
*************************************
Maze: 1
File offset: 148C, DS:00FC, table offset: 0000
Size X: 22
Size Y: 11
End: 14BD
Length: 50
XXXXX
X X
X* X
XXX *XXX
X * * X
XXX X XXX X XXXXXX
X X XXX XXXXXXX ..X
X * * ..X
XXXXX XXXX X@XXXX ..X
X XXX XXXXXX
XXXXXXXX
*************************************
All the map are available.
The maps are compressed with a Huffman-like approach using bit sequences of variable length. Each level has the following structure:
1 D3 D2 D1
, where the number of repetitions
N = 2 + D3*4 + D2*2 + D1
(values 2 to 9). <CODE> has five different
values: 00 - an empty space, 01 - a wall, 10 - a barrel, 101 - a place
for the barrel, 111 - a barrel already in place.And so for all 60 maps.
In the file pushermaps.c there is an implementation of the simple decompressor.
When disassembling the maps also had been presented in the convenient plain text form but still compressed.
level_01 db 16h, 0Bh, 0A2h, 0DFh, 38h, 32h, 1Fh, 38h, 2Ah, 3, 0E6h
db 12h, 0C0h, 0A5h, 0F2h, 83h, 2, 81h, 3, 0E4h, 12h, 82h
db 25h, 6, 0CDh, 64h, 22h, 51h, 0ACh, 11h, 0A1h, 0Ah, 5
db 0E5h, 11h, 0B1h, 14h, 82h, 29h, 82h, 31h, 0A0h, 0E1h
db 2Ch, 18h, 0D1h, 0CFh, 80h, 0Ch, 8
level_02 db 0Eh, 0Ah, 0F6h, 58h, 0Ch, 68h, 0Dh, 94h, 0C6h, 80h
db 85h, 2, 82h, 18h, 0D0h, 15h, 4Ch, 10h, 0C6h, 0C2h, 18h
db 21h, 8Dh, 1, 6, 4, 39h, 10h, 0A0h, 81h, 80h, 85h, 2
db 8, 20h, 60h, 34h, 1Bh, 0Ch, 1Eh, 0CAh, 7, 4
level_03 db 11h, 0Ah, 0E3h, 9Fh, 0Eh, 7, 0C2h, 11h, 42h, 1Fh, 8
db 50h, 23h, 0E0h, 85h, 4, 0Ch, 1Eh, 84h, 8, 0A6h, 0B4h
db 10h, 85h, 2, 82h, 59h, 0D4h, 28h, 14h, 90h, 0D6h, 83h
db 0DFh, 7Ch, 0Eh, 1
… etc.
So if you’d like to implement a quick and compact Sokoban for some reason, there is a bunch of already created and nicely compressed maps.
I know that there are tons of levels for Sokoban in the internet, plus automated Sokoban solvers and similar stuff. But it still doesn’t make all that fun of dissecting the twenty years old or more binary less interesting.
The project is available at GitHub’e – https://github.com/begoon/sokoban-maps.
(photo from Wikipedia)
Now it is genuine!
Open it up! It didn’t risk to detach the keyboard, so I had to photograph from two angles.
The label “1982 ISSUE TWO” is visible at the bottom left.
There are a few lovely “extra” parts stuck on some chips. Maybe it was a result of the “last minute tuning” or maybe the previous owner did it.
The seller also gave a bunch of tapes. I haven’t seen 15 minutes cassettes before.
Extras.
So lovely books.
People had guts to put proper illustrations in manuals at that time.
I also bought a very old TV at car boot sale for a pound.
Unfortunately, this machine didn’t boot up. It beeped and printed rubbish on the screen. The seller argued for a while saying something about video synchronization problems but after the video (below) agreed to refund. Reluctantly I had to send it all back.
Programs should always have the form of paragraphs of comments that describe the intention of the program followed by paragraphs of code that implement that intention. All of the formatting should be designed to make readers as able as possible to read the code easily; the compiler doesn’t care. In particular, follow conventions of mathematics and your native language, not those you found in some random language manual. Write the comments first and then write the code, not the other way around. If you don’t know what you want to achieve and why, any code you write is, by definition, incorrect.
“Etudes” is very different from other computer science and programming textbooks. “Etudes” covers most of the important areas in a quite an unusual way. How did you come up with the idea to collect realistic problems from various areas and put them in a play situation? Could you tell us a little story about how and why the “Etudes” were born?
From 1974 to 1979, I was teaching at UC Davis. There was a new Computer Science graduate group there formed to give degrees in Computer Science, but it was really just a loose collection of faculty members from several departments. The department of Applied Science hired me as the first professor with a computer science degree to pull the program together. Along with a small group of other professors and adjunct instructors, we designed a new curriculum, put degree programs in place, and began teaching a number of new or redesigned courses.
During this redesign, I got the other faculty to agree that students needed to know how to work in the real world as well as to learn the theory surrounding computer science. By “work” in the real world, I don’t mean that I wanted to teach them “job relevant” topics; in my experience, such knowledge becomes obsolete quickly. Rather, I wanted to have them work on problems that included making imprecise specifications precise, on problems whose requirements were unclear, on problems that had real world application, on problems that required working with other people. For example, in the compiler construction course, students were required to work in two or three person teams and the same grade was assigned to everybody on a team. Part of the assignment was to write up how they divided their work and whether they thought the division was successful.
After a bit, we decided to teach a programming projects course to consolidate student knowledge of how to build complete applications. We thought of a few projects and I wrote them up. But when I looked around for more, I could only find one book that had only “toy” exercises in it – for example, print a table of all the possible binary Boolean operators. I also found one or two problems that Carnegie-Mellon used in one of its courses. So before long, I was writing “Etudes”. As it turned out, I left UC Davis soon after the book was published, so I don’t really know if they continued to use it in the projects course.
How long did it take to write the book?
“Etudes” took about 18-24 months start to finish. I had written up a few chapters for course work already. Contract negotiation was very fast, as I remember; perhaps only a month or so. I spent about a year writing the remaining chapters. Of course, I had to fit that around my other work. Then there was perhaps six months of review and revision. Once everything was in place, I only had to wait a month or two after final galley proofs went in to see my first copy.
Why didn’t you publish any other books after “Etudes”? Why wasn’t “Etudes” republished?
I left UC Davis to go work at Bell Labs. While I was there, I ran a project that built one of the first Ada compilers. We published technical papers, but there wasn’t a lot of time to write another book. After Bell Labs, I came out to Silicon Valley for a startup and, again, there wasn’t a lot of time for book writing. I also started a family and taking care of kids is (as most parents know) fairly time-consuming. And then you know how it goes.
Back in the early 70’s, Juris Hartmanis (one of my favorite teachers) told me that if you wrote a computer science textbook, you would make enough money to buy a new car. The reason was that there were at the time enough libraries who would all buy a copy of the book that the minimum income would be enough to buy at least a Volkswagen (VW Bugs were still around then!). If you got lucky, you might be able to buy a Mercedes Benz. And, of course, if you wrote something like Samuelson’s “Economics”, you could buy a new Lamborghini every year.
But for textbooks to do well, they have to be adopted in courses. Unfortunately, only a few colleges in the US adopted “Etudes”. So there wasn’t a lot of income from it. And the publisher had no real incentive to publish a new edition. There were a lot of individual people who bought copies, but that doesn’t help for textbooks.
One odd thing did happen, though. Right after “Etudes” was published, the Soviet academic publishing office reviewed it and decided to buy the copyright so that it could publish “Etudes” in Russia and the other Soviet Union states. This was a real diplomatic breakthrough, actually; quite commonly, the Soviet Union had simply ignored copyrights in the past. The payment from the Soviet purchase was a pleasant surprise.
Unfortunately, though, not a very profitable one. They paid $1,200 for the rights and that wasn’t a lot of money even in the late 70’s. I got half or $600. The good part was that “Etudes” was translated into Russian and used widely in the Soviet computer science curriculum. I work, for example, with two Russians who used it as undergraduates. Alexander, you are a third! I think that there were at least 60,000 Russian copies published.
And what kind of car did I buy? Actually, I bought a brand new piano that’s still sitting in my living room.
How did you find topics for the etudes?
There were several ways. One or two were problems I had in practical life. For example, the gasoline mileage etude and the investment etude came from things I worked on for myself. Some came from my interest in playing games with computers and ideas in artificial intelligence; obviously, the kalah and the management game were of that sort. The language problems at the end were revisions of the materials from our compiler course. Some of the problems are straight computer science just dressed up: the Turing machine, the map coloring, the format scanner.
Chapter 4 on automatic text formatting was inspired by a colleague at Lawrence Livermore Laboratory. He had written one of the very first automatic formatters and I had used it when writing my dissertation and the drafts for “Etudes”. Remember, this was before Unix was well-known and even “nroff” was pretty fancy stuff. So a problem that introduced the topic seemed reasonable.
The problem solutions were just those that “fit” within the production confines of the book. And I wanted students to see what “real” code looks like. Notice that the last problem even says that the solution is incomplete.
For each etude in the book you recommend how long it should take. Since 1978 programming languages have been “slightly” changed and become “a little bit” more powerful. But even nowadays, your timings recommendations look quite aggressive. Did you “play” the etudes yourself working over the book? Do you have your favorite one?
In retrospect, this was a big flaw. The durations are aggressive even now, even with very good languages like Ruby (my current favorite) that would do a lot of the work for you. Remember that when “Etudes” was written, Unix was just becoming broadly available, Basic, Fortran, and Pascal were probably the most common teaching languages, and C was only just becoming known. There were essentially no terminals to program on (although my students actually did mostly have terminals because of their associations with Lawrence Livermore Laboratory) so that students still prepared program card decks.
I’m not sure I have a favorite etude. I like the TRAC interpreter. It teaches a lot about the fundamental topics of programming languages, topics that are often ignored in course work. As “Etudes” mentions, TRAC and Strachey’s GPM are languages that were invented simultaneously and independently; the two author’s published papers appeared within a few months of one another. The two languages are very similar and both show how a focus on text manipulation can solve large problems. It is particularly rewarding to study them together and to understand how they agree and differ. I also like the EC loader etude because I think it solves problems in programming language translation that are still common; why techniques like this are not still in use is beyond me.
I did not actually get a chance to do a lot of problems when I was writing the book. However, when I went to work at Bell Labs, I had to teach myself Unix and all its tools. So I decided to try programming the gas mileage etude in “awk”. Oddly enough, I discovered a bug in “awk” itself that I reported to Brian Kernighan. He commented that he knew there was a problem, but nobody else had been able to pin it down. The bug turned out to be a computer science classic: “awk” was using some allocated memory after it had already been freed.
I cannot skip a question about the etude about the Vigenere cipher. There were a couple of setups in that chapter, like a “tiny” typo in the form of a missing line in the crypto text, and also a slightly wrong hint about how to crack the cipher. Of course, it did not stop people from cracking it at all, and it turned out that those “accidental complications” made this etude even more attractive. Could you shed some light on what happened there?
This is embarrassing. As it turns out, the publisher reset all the copy text before the book was printed, but I supplied the illustrations directly. Unfortunately, I supplied a bad illustration for the actual code text to crack. It wasn’t until the Russian translation that anybody noticed (or, at least, told me about it). Any lack of clarity in the text itself was just because of my writing. If that is the only problem in the text of the book itself, I would be very happy.
I presume that the preparation and maybe solving the problems in “Etudes” was not a straightforward process. Were there other “gotchas” or Easter Eggs in “Etudes”?
There are no other intentional “gotchas”. When I tried the mileage problem, I realized that it might not be as well specified as it might be. There are probably some other places that could be clarified.
Actually, writing was not too bad. Of course, making the solved problems be suitable took a lot more work than a student might put in; I didn’t want to look silly to the readers.
Fortunately, I had a lot of help and that made a big difference. The text and the illustrations were originally produced using innovative formatting and drawing programs developed by Hank Moll and John Beatty. This made the work much easier. I wrote the book itself on “coding sheets”, paper specially marked to make it easy to keypunch cards. The sheets then went to our keypunch team who actually did a lot of proofreading as they went along. When they thought I had made a mistake (or had made an outrageous pun), they would punch both an original and a correction card so that I could choose which was correct.
My publisher asked several outside reviewers to read and comment on the book drafts. One reviewer of the complete book draft made comments on almost every line. He made the book much better than it would otherwise have been. At the time, I didn’t know who this reviewer was. Later, I found out it was Steve Muchnick (well known for his compiler optimization book). I cannot thank him enough.
If you write “Etudes” today, which topics would you add?
Digital signatures and public key cryptography protocols would probably be a new topic. I would probably tune up the multiple precision arithmetic chapter. I would almost certainly change the compiler/language chapters to include more modern ideas. One thought (due originally to John Fletcher) would be to have a pair of etudes to write a Lisp interpreter in Ruby (maybe) and a simple Ruby system in Lisp. Or something else of the sort.
I would probably drop one or two of the simpler etudes. There is a lot of new artificial intelligence problems (particularly in genetic programming) that would be suitable. There are new ideas in data structures that would be fun to explore (how much better are red-black trees than simple-minded binary trees in terms of performance?). Honestly, though, I can’t tell what would make a good problem until I work on it for a while.
Most of the etudes were about implementation rather than figuring out a way to attack a problem in the first place, because you almost always gave quite solid background and enough details of a solution. Do you believe that it is also important to give students problems without even a clue of the solution? How such problems may look like? And how would you measure a degree of success of the solution?
In the real world, you are always faced with some problem to solve and the first issue is figure out what the problem really is. It is good to train students to do that analysis – and in the real world, that skill is rare – but that wasn’t really the idea in “Etudes”. That would be a course in itself. The point of “Etudes” was to give students a reasonable problem setup and to have them practice full application development. There is still a lot of room for student choices. For example, the solved problems make points about the quality of the code, the presentation of results to humans, and the performance of the final application.
There is a probably a course with problems like you propose, but then the course lectures would have to teach analysis techniques and the book itself would have to include materials on analysis. That’s a different book.
Charles, regarding your interests and work apart from “Etudes”, I know your background is compilers. For instance, you developed a compiler for the XPL language. What do you currently work on at Oracle? Still hands on programming?
I am still a programmer. One of the things I like about computer science is that you can actually build the things you think about.
My basic background has always been the theory, design, and implementation of programming languages. When I was an undergraduate, computer science didn’t exist yet so my undergrad degree is in applied mathematics. But my undergraduate thesis was on adding new features to TRAC (does that remind you of one of the “Etudes”?). I have been responsible for XPL, for a complete Ada compiler system, for the LR parser generator (with Al Shannon), for a proposal of array operations in Fortran, for a new dataflow language (with Jim McGraw), and (more recently) for the new code generator and optimizer in the Oracle programming language PL/SQL. Lately I have been working on some special projects at Oracle in the area of programming languages. If they work out, you’ll see the papers on them.
Looking at the topics in your book, it is not easy to guess which area of computer science is your favorite one. Even in the section about compilers you offered an interactive functional TRAC along with a regular procedural language. Do you have favorite areas of computer science?
My favorite computer science topics are:
Programming languages, particularly the specification of their semantics.
Data structures and algorithms and their performance.
Games and artificial intelligence.
In practice, my professional career has mostly been spent on the first topic. I have spent a lot of time working on compiler/language/application performance. This is the second topic, of course. When I was in middle school, I was already interested in computers and I already liked games. That’s where the third interest comes from. For me, computers are the world’s greatest toys and always have been. But I won’t see all the advances that my younger son with his iPad now takes for granted.
You can see the outgrowth of my interest in computer games in a couple of papers I wrote (with some friends) about Kriegspiel, a variant of chess. The papers are published in the “Computer Journal”. One of these is a complete program to check the legality of moves in chess and Kriegspiel. At the time the paper was written, there were a few example programs for students, but these were not very careful and missed some of the more subtle rules of chess. There were, of course, chess programs available, but this was long before open source software and at a time when a few chess programmers were trying to make a living selling chess products. So they obviously didn’t publish their secrets. So far as I know, this was the first published paper to provide a complete rules checker and I haven’t seen one since; it may still be the only refereed publication of a rules checker. You can find an audio interview about the project at the Computer Museum here in Silicon Valley.
My approach to computer science is generally to see if I can find some good theoretical model for a problem and then understand how to apply that in practice.
Over the years, have you met that programming language which you liked most for some reason?
Right now, I like Ruby a lot. I have written great gobs of C, but I don’t like its lack of security. Back in the day, I liked XPL which seemed like a reasonable version of C (of course, it was developed independently) with much more safety.
When I was a grad student, I wrote one long Snobol program. I used it to model some research that had been done in code optimization. It was possible to turn constructive proofs of theorems directly into Snobol routines. That was eye-opening. It has made me partial to languages that have interesting inference mechanisms.
I’d like to have a chance to learn and use Haskell or one of its friends for a good project. I think it would teach me a great deal. I should note that writing a toy program in a language is not the same as knowing the language. I have been doing Ruby for a year now on one project and I still learn something new every day.
When languages like C were invented, people wanted to be close to the metal. Understanding how the metal worked was the must for any professional programmer, and as a consequence, it dictated a manner of teaching programming. How would you teach programming nowadays? Not computer science in general but hands on programming?
I never understood the focus on hardware although I started doing assembly language on a machine that didn’t even have mnemonic op codes in the first assembler I used.
Actually, I think you asked the wrong question. The problem is not “programming”, but problem analysis and application development. You can program anything in any language; the question is whether the language is a good “human” fit to the problem and whether or not it is efficient enough.
Thus, I would teach principles of programming languages and then ask students to try and understand those principles in each of a number of languages. For example, did you know that Fortran always includes an interpreter? (For format statements, of course!) And what’s more, in Fortran 77, you need to implement call-by-name as part of that interpreter. You also probably want to use exceptions to build your interpreter. So the basic concepts pop up in surprising places.
(By the way, that tells you why the formatting etude is in the book. Over the years, I built two separate Fortran format interpreters.)
We started this interview from the book. Did you have your own “version” of “Etudes”? Any books and people significantly influenced on you as a programmer and scientist?
When I was in high school, a friend of my parents helped me get a summer job. That job turned into work at the local school system’s data processing center. In those days, data processing meant cards, card sorters, card punchs, and the like. But the next summer, the school system bought an IBM 1401. Because nobody else knew more than me about it, my supervisor let me be one of the programmers. Within a few weeks, I was writing payroll programs for a large public system.
This experience helped me get a part-time job in college also working with 1401’s. From then on, my life was working on computers and I decided to go to grad school in computer science instead of getting a full-time job. With more good luck than prior knowledge, I went to Cornell. The faculty there was incredibly helpful. Probably the two faculty members who taught me the most were Juris Hartmanis and John Hopcroft. I still think that Hopcroft’s language theory book is a classic (in the first or second editions). But everybody on the faculty there was extremely concerned that their students did well. And, if you look at the Cornell graduates from the late 60’s through the mid-70’s, you’ll see a really wide range of talent went through the program.
So far as books go, my favorites are probably those by Knuth. I really recommend reading the “Art of Computer Programming”. You won’t understand most of it the first time. My theory is that you read it over and over, a few pages at a time. Each time you look at it, a little more sinks in.
I also read all of Newman’s “The World of Mathematics” which I first saw in the 10th grade. This gave me a lot of historical background in math and mentioned enough about computers that I was helped along. I think this is still a useful introduction for young people (not to mention old people!).
Everybody comes to the world of computers differently. Maybe nowadays it is much easier because computers are everywhere. How did it happen that you chose computers as your profession in the days of mainframes and punch cards?
Well, some of the answer is in the previous question. My father owned a small hand-crank desk calculator with a design similar to that described by Felix Klein in his mathematical education book. My father’s came from the Orient (he got it during WWII, I assume), had no instructions, and was covered in Japanese symbols – but fortunately Arabic numerals. I spent many hours figuring it out.
When I was in middle school, I already liked computers and mathematics. Sometime along in there I got a toy “computer” that was made out masonite, old-fashion clasp paper clips, screws, light bulbs, batteries, and wires. It was really what would be called a gate simulator today, but you could build some very simple circuits with switched inputs and lit-up light bulbs as outputs. I was fascinated. I also found some other books about early computers and they were very interesting to me.
In my first year of high school, I went to NSF summer camp where we took some college course in algebra and group theory, learned (well, in my case, was mystified by) some finite differences, and got to program a computer. I also learned to play bridge because two of girls wanted to have a fourth (when you are in high school, girls are a powerful incentive!).
So by the time I got out of college, between the experiences I have described and some college courses (notably with Tom Cheatham), I was pretty much hooked.
Wrapping up, there is a saying about the three things a man must do. Can you name your three things a programmer must (or should) implement?
I’m not sure I have three things that I think programmers should implement. But I know some things that I think every programmer should either know or be able to do.
Some ability to do formal mathematics. The level necessary to understand Hopcroft and Ullman combined with a little graph theory is enough. Discrete mathematics is essential; calculus is only necessary for folks working in special application areas.
Some ability to write clearly in their native language. Dijkstra said that if a person couldn’t write their own language, they couldn’t write a correct program. (Well, that’s what I hope Dijkstra said!). Writing programs is essentially the same as writing a non-fiction essay. If you can’t make the step-by-step connections clearly in your own language, why would we think you could in C, for example?
Remember that a program is primarily for communication with humans, not computers. When you write a program, the computer will do whatever you say. You must convince the humans who read the program that what you have asked the computer to do is the correct thing to do. Remember, the computer doesn’t care about correctness.
(Bonus answer – 4 for the price of 3). The answer to the last question means that programs should always have the form of paragraphs of comments that describe the intention of the program followed by paragraphs of code that implement that intention. All of the formatting should be designed to make readers as able as possible to read the code easily; the compiler doesn’t care. In particular, follow conventions of mathematics and your native language, not those you found in some random language manual. Write the comments first and then write the code, not the other way around. If you don’t know what you want to achieve and why, any code you write is, by definition, incorrect.
If you look at the last “Etudes” chapters, I hope that you can see that even back then, I was trying to apply these principles.
Thanks, Charles for the interview answers. We (fans of “Etudes”) still hope for your new books or other publications and wish you all the best. In meanwhile, if you, my dear readers, have not seen “Etudes” so far, please grab a copy and take a look. It is worth it.
■
// Charles Wetherell, Alexander Demin
// August 2012
P.S. If you have questions to Charles, please, put them in comments below.
For instance, to form a stub number for a 16- or 32-bit value, people usually use words from the following set:
ABBE
ACED
BABE
BADE
BEAD
BEEF
BODE
CADE
CAFE
CEDE
COCA
CODA
CODE
DACE
DADO
DEAD
DEAF
DECO
DEED
DODO
FACE
FADE
FEED
FOOD
OBOE
For example:
DEADFACE
ACEDFEED
If we don’t care about the symmetrical split, the vocabulary is a bit wider:
ABBE
ABE
ABODE
ACCEDE
ACCEDED
ACE
ACED
ADD
ADDED
ADO
ADOBE
BAA
BABE
BAD
BADE
BAOBAB
BE
BEAD
BEADED
BED
BEDDED
BEE
BEEF
BOA
BOB
BOBBED
BODE
BODED
BOO
BOOBOO
BOOED
CAB
CACAO
CAD
CADE
CAFE
CEDE
CEDED
COB
COCA
COCOA
COD
CODA
CODE
CODED
COFFEE
COO
COOED
DAB
DABBED
DACE
DAD
DADO
DEAD
DEAF
DEB
DECADE
DECAF
DECO
DECODE
DECODED
DEE
DEED
DEFACE
DEFACED
DO
DOC
DODO
DOE
DOFFED
EBB
EBBED
EFFACE
EFFACED
FAB
FACADE
FACE
FACED
FAD
FADE
FADED
FED
FEE
FEED
FOB
FOBBED
FOE
FOOD
OAF
OBOE
ODD
ODE
OF
OFF
For example:
BADCOCOA
BADCODED
FADEDDOC
CODEDBOB
On 64-bit systems we can create almost complete sentences.
EBBEDDEADBAOBAB
Does anybody know what to say to the next, 128-bit generation?
In short – it is awesome! I’m not talking about the material they provide. It is great without any concerns. I’m talking about the technology of online education.
Briefly. The course lasts six weeks. Each week we are given video lectures, a hour and a half in total, broken down into 10-20 minutes episodes. The videos are interactive, and sometimes you’ll get a quiz along with watching. For each episode there are two PDFs. The first has live commentaries written by hand. They appear on videos along with lector’s voice. The second is a regular PDF with slides. You can watch videos as many times you want: in the browser or downloading it. English subtitle are available. There is a cool feature in the online video viewer – you can change the speed from 0.5 to 1.5, so it turns to too much “blah-blah”, there is a way to speed it up.
Also you’re given with a list theoretical questions and one programming question each week. You only have a few attempts for submission. The best answers count.
There is no real deadline but submissions after the “official” deadline will be graded with 50% “discount”. So, if you’re up the certificate after the course, it’s worth doing it in time.
At the very end there is a final exam. You’ll have 3 hours to complete.
There are forums available where you can communicate with other students and course owners.
Subjectively, I’d make the course a bit “tougher”, but in general it has perfect balance for a couple of evenings a week.
Conslusion: Strongly recommended. Not this particular course because everybody has different interests, but the technology in general.
Nowaday if someone says that he or she has no money for an expensive and famous university, you can recommend to save little money for internet and an inexpensive computer. That’s it.
I’m up to try 6.002x: Circuits and Electronics next. Just curious how they do a practical part for electronics online.
From 64MB to 32GB.
An idea occured to join them all together into RAID, for example, a concatenated disk. In OSX it can be done in a few clicks. The idea is pretty silly but it was interesting to try.
I’ve bought an extender with 12 ports.
Satechi 12 Port USB Hub with Power Adapter & 2 Control Switches
Plugging them all in…
Start the Disk Utility and check that all the sticks are recognized.
Create a “concatenated disk” named “Crazy RAID” and add the sticks into there.
Confirming…
Wait for five minutes and it’s done.
“df” perfectly sees the drive.
Now we can copy something there and check the transfer rate.
Of course, this is just a toy without real usage. All the sticks have different capacity, so the concatenated disk is the only choice in terms of the RAID features. Write speed depends on a particular stick which is currently in use. On concurrent read we may gain some speedup.
But it looks cool!
Below there is a bunch of photos of things which had caught my eye. Of course, there is much more stuff in the museum.
The real BBS, even if running Windows.
The ancestors of an era.
The NeXT CUBE.
Brothers! This is an 1200 bps modem, or 300 full duplex.
Have you seen The Matrix? Remember the modem they used for teleportation?
I’d be surprised if there isn’t Raspberry Pi in the British museum, so here it is.
A portable laser-disc storage.
Oh, I should probably buy this one and relax.
Brothers! Fasten your seat belts. This is the whole room of working BBC Micros. Books and manuals are ready to help. You can sit and program.
And I did.
BBC Micro inside.
Books are everywhere, new ones, old ones.
RML 380Z
PET Commodore
ELIZA can ask questions and answer.
The history of computer memory, from bulbs to SIMMs, DIMMs and all the rest of it.
A flash drive from the past.
Computer trainers. I love these charmy devices.
Would you like to study on this one? I would.
Ergonomic keyboards weren’t invented yesterday.
A very expensive computer for business from the 80-th.
There are a few lovely cassettes on a huge bookshelf with manuals. It turns out that they were used not only at home.
Go to a middle heavy weight. Now floor-standing machines instead of desktops.
Cray
PDP-8
PDP-11
A XEROX workstation.
VAX. I “saw” this guy only via a terminal when I was studying.
A graphical station.
Terminals, terminals.
A hall of epoch-making computers.
Recognize this guy?
In the corner? We (Russians) knew it as a Zonov’s clone implemented on Soviet parts, but this is the original.
In action.
It turns out that there were hell a lot of machines in the keyboard case form factor.
This computer had launched Microsoft.
This one had started Apple.
You can land your bottom on the bench and play rare games of epoch-making rare computers form the 80-th.
A room to play the Flight Simulator.
The mainframes.
UNIX. The Beginning.
A real radar monitor.
By the way, a true computer museum cannot be without WiFi, so here it is.
PDP-11
An analog computer. In digital computers we break down tasks into primitive binary operations. In analog computers we have to use operational amplifiers to add, subtract, integrate or differentiate electrical signals.
Convenient, however.
A new, not yet complete exposition.
But something is already there.
Punch card machines.
A (attention) calculator!
The archives of Computer Weekly.
Brothers! This is just awesome, the must see place for computer enthusiasts. Unfortunately, I had only forty minutes for everything, so maybe I’ll go there once again.
This device allows to connect legacy RS-232 devices to the world of TCP/IP. We used such things years ago developing software for POSes and ATMs. Just change a dialing script from something like ATDT12345678
to ATD192.168.1.1
and off you go.
Have no idea what to do with this now. I even have no proper RS-232 port, only via a USB cable. Interestingly, back those days such devices were very expensive.
Programming 32-bit Microcontrollers in C: Exploring the PIC32 (Embedded Technology)
I stopped programming for PICs about ten year ago when we were developing a GSM phone based alarm system. We used the PIC16 chip and a DTFM decoder (be the way, the feature of controlling the alarm system dialing to it and using DTFM signals was pretty awesome that time) and a bunch of sensors.
I also worked with PIC12. I was stunned of just an 8-pin chip programmable in C!
Then I moved to the UK and left a lot of stuff at home. But recently it all has come back when I built Maximite and also bought Raspberry Pi. Since then the technology has stepped forward, of course. Now PIC32 chips are based on the MIPS kernel, almost fully 32-bit and have a primitive virtual memory capabilities. It allows creating simple operating systems when the kernel has its own protected memory from users processes. For example, RetroBSD perfectly works on Maximite.
Back to the book. I wanted to refresh my PIC knowledge and opened the book. To my surprise I swallowed it in a couple of evenings non-stop.
The book is organized perfectly and suitable for people having minimal experience in microcontrollers. But it covers serious topics such that:
All example are complete and working. It can be even better if you have a companion CD of the book.
The only little issue was that the author used MPLAB and the C32 compiler (remember, the book was released in 2008), but Microchip strongly recommend migrating to MPLAB-X (which, by the way, the only choice for Linux and OSX) and the XC32 compiler based on GCC. Of course, the project files can be easily converted to the new MPLAB-X format, the book lacks examples of simple Makefiles. But it compensates it with exhaustive screenshots and commentaries about configuration windows, the debugger and the simulator. It is possible to read the book even without a computer.
If you read the book, you’ll understand what can be implemented with PIC32 and how things work in it. If you decide to implement a project, you have enough examples plus understanding how to start the crystal.
The conclusion: Highly recommended.
When I found that the PIC32 chip could be made to use a VGA display and keyboard, it was only natural that I should try and recreate one of the earlier computers that I had so much fun with in my youth.
Geoff built a microcomputer, Maximite, using a PIC32 chip. Maximite is a complete appliance utilizing modern interfaces. It uses VGA for video, PS/2 for the keyboard, SD cards as storage, USB for communicating, and it runs BASIC! Doesn’t it remind you something? In my view, Maximite is a perfect tool to study microelectronics allowing to touch how hardware meets with software.
Today we can ask Geoff a few questions in person. Geoff lives in the suburb of Kensington in Perth, Western Australia. Perth is a long way from anywhere but with the wonders of the Internet and FedEx the world is a much smaller place now, so it doesn’t stop Geoff to design and produce great hardware and software.
Hi Geoff, thanks for the interview. Let’s begin with a light question – why “Maximite”? How did you invent this name?
At the beginning of the project I wanted to call it The Mighty Mite meaning something that was small (mite) that was also powerful (mighty). When the design came to be published in Silicon Chip magazine the editor discovered that the name had been trademarked by a food manufacturer, so he suggested Maximite as an alternative. By then I could think of nothing better, so that became the name. As it turned out, that is also the name of an explosive which, in a strange sort of way, is also appropriate.
Maximite is very popular. Do you know at least rough figures of how many of them were built? Also when and how did you realise its incredible popularity?
My standard answer is that “thousands are in use” but I believe that a more accurate number is somewhere between 3000 and 4000. About half of them have been built by people with a soldering iron and the other half purchased as an assembled board from companies who make copies of the design.
I realised that it would be popular when I looked at the number of visitors to my website, in the first few days after the magazine article appeared the number of unique visitors to my website jumped by 100 times before settling down to a long term increase of 20 times the pre Maximite days. What was interesting was that my URL did not appear in the first magazine article so many people read the article and immediately used Google to find out more.
When I started on the design of the Maximite I was thinking of the computers of the 80’s which provided me with a lot of fun with back then. They were very popular in their day and, as the modern computers have become more complicated and difficult to use, there was a vacuum left behind. I believe that the key to the success of the Maximite was that it filled that space.
For Maximite you had to build hardware and software. This is quite an unusual position nowadays for people working with computers. Professionals in software rarely look at hardware and vice versa. Where are you here? Are you more a hardware engineer by nature or a programmer?
Yes, that is one of the things that I have found since the Maximite design was published. People tend to look at it through the eyes of a hardware engineer and do not appreciate the software - they want to add electronic features without regard to the firmware or how the BASIC language could accommodate them. There is always the reverse too, people who only think in terms of the software. I think that I am best described as a programmer but with a strong electronic background.
Another item that is often missed when people try to change the Maximite is its “look and feel” (ie, how the user uses it). This must be consistent with the hardware and software. All three are part of the integrated whole and this is often not appreciated by either the hardware or software specialists.
I personally was amazed of how solid and complete the Maximite project is. You designed the board, the case, software, and the kits of Maximite are available online. In the world of individual homebrew projects people usually don’t tend to finish off their projects fully to such a polished shape because they loose interest. How did you manage that?
This primarily came from my desire to have the project published in Silicon Chip magazine. Most of their projects are a complete “product” designed by a professional engineer and I had to meet that standard if I wanted them to publish the Maximite project. This attention to detail is unusual today as magazines tend to only half finish the project, things like documentation, a box to put it in, a nice front panel and so on are often ignored.
I find that the discipline of designing projects for the magazine is quite good. I have a number of personal projects (ie, ones not published by the magazine) that are still sitting around today in a half completed form because I do not have that discipline operating on me.
Geoff, you started your microelectronics hobby when you retired. How much time a day you usually spend on it? Again, you delivered the Maximite project in a such complete form, so seems it required some time to put everything in place.
Yes, I have spent far too much time on it since I have retired, but
interestingly it was something that I had to do. It is as if there
was a design engineer hiding inside of me that could not get out while
I was following my professional career (which was not in electronics).
The moment I retired he broke forth.
With the Maximite things seem to come in bursts, I would spend twelve hours a day for a month or two then have a break for a few weeks. It is good that I enjoy working on projects like this because they do take up a lot of time. I estimate that the Maximite has taken about nine months of solid eight hours a day, forty hours a week to get it to where it is now.
MMBasic. I know you struggled to find software for Maximite. A few existing implementations of BASIC you tried didn’t really fit, and you ended up with your own implementation of BASIC. Could clarify the licensing and distribution model of MMBasic? Any plans, maybe, to develop a Maximite framework or a library allowing to deal with Maximite peripherals in the form of device drivers? It could be useful in developing alternative firmware for Maximite, for example, running Lua or RetroBSD.
Yes, when I started on the Maximite project my hope was to find a free version of BASIC in the Internet and use that. But I found that the good implementations of the language were not available freely or had some other issue. So I decided to write my own.
Initially I released my version of BASIC (called MMBasic) under the GPL GNU licence but I had a bad experience with one commercial organisation who removed my name from the software and claimed it as theirs, they even changed the copyright to their name. So I created my own license where the software is freely available but it cannot be redistributed without my agreement.
This has worked out very well, a number of universities and commercial organisations are using it and I have an agreement with them covering what they will do with it. This is much better than allowing someone to anonymously download my software and then see it appear on the Internet with their name and copyright.
At this time I do not have plans to develop a loadable device driver
type of architecture. The main reason is that the PIC32 chip does not
have enough memory to do all the things that it already does (VGA,
keyboard, USB, etc) and implement a loadable device driver system.
There are more powerful chips around but they are impossible to hand
solder so I have stayed with the PIC32. Products like the Raspberry
Pi can have loadable device drivers but they also have a complex
operating system - just what I was trying to avoid with the Maximite.
Let’s move a bit to your personal background. When and how did you study to solder, for instance? I know many people looking for some experience in electronics but they are afraid of soldering seeing it potentially difficult or even impossible. How would you recommend people starting messing around with electronics hands-on?
I cannot remember when I first started soldering but it was probably around age 12. I am probably the wrong person to ask but I cannot see the problem with soldering – you only need a cheap soldering iron and some wire solder.
One of the great things with electronics is that it can be a very cheap hobby. I would recommend starting with a simple and cheap project like flashing a light or adding a clock chip to the Arduino and just buy the bits including the soldering iron. You might make a mess of the first project but that is not important and by the time you have reached the third project you will regard yourself as almost professional.
Would you recommend any books or websites where beginners could make their first easy steps into the wonderful world in electronics? What kind of minimal level of knowledge could be enough to design and build, for example, Maximite?
My recommendation would be to start with a magazine subscription. There is Silicon Chip (in Australia), Elektor (in Europe) and Nuts and Volts (in the USA). All of these will accept international subscriptions. If you are starting with no knowledge at all probably Nuts and Volts would be the best. When you subscribe to a magazine you will have a constant stream of ideas and projects to tempt you into trying something different and fun. So a subscription is well worth the cost.
The level of knowledge required to build the Maximite is not great because it comes already designed and with working software. On the other hand, to design it you would need a great amount of expertise. I have a lot of expensive test gear, I trained as an electronics engineer in my early days and I have about 30 years of experience in systems programming - all that is necessary to create something like the Maximite.
Now we have plenty websites providing a marketplace for individual hardware developers to sell their ideas and designs, for example, SparkFun.com. The internet allows people to get some money from hobby. You, as a person, whose hardware designs are being wildly sold over the internet, do you think it is feasible to live on income from such business? or volumes are still minor?
It would be very hard to make a living income from that sort of thing. There are a few, like the designers of the Arduino, who are probably doing OK but there are many more who have to keep another job so that they can pay the rent. Part of the problem is people like me who produce sophisticated designs and then give them away for free, it is hard to compete with that and your customers come to expect products or software as a very low price.
A bit of personal stuff. Can you name any people, for example, engineers or programmers, which you did or would like to learn from?
The programmer who was the most influential on me is Brian Kernighan who, along with Dennis Ritchie, designed the C programming language and wrote a book on it called “The C Programming Language” in 1978. Even today that book is a wonderful introduction to the C programming language and well worth a read if you want to get into programming in C. When I write my magazine articles and web pages I always remember his clear and easy writing style and try to emulate it in what I do.
Probably the most influential engineer is Jim Rowe who was creating projects and writing for electronics magazines in Australia when I was a young teenager. He is still doing it today 50 years later and I always find his projects well designed and his magazine articles clear and easy to understand. He is someone else who I try to emulate in my projects and articles.
By the way, how did you come to the world of computers and engineering? What was your first computer or programming language?
This will date me somewhat (I am 64 years old). I remember the Altair computer when it was first released and a little later the company I was working for became the distributor for Intel in Australia. I was studying as an electronics engineer at the time and my first computer was one that I hand soldered using the Intel 8080 chip and 512 bytes of RAM (yes bytes).
Back then BASIC was the only language that you could use because of the limited memory capacity and hardware. Much later, when we had disk drives, you could use more sophisticated languages and in my time I have written programs in most of them (Fortran, COBOL, PL/1, assembler, Pascal, etc).
As you can see, I was involved in electronics, small computers and BASIC a long time ago. So, when I found that the PIC32 chip could be made to use a VGA display and keyboard, it was only natural that I should try and recreate one of the earlier computers that I had so much fun with in my youth.
Wrapping up, I’d like to ask you to shed some light on your future projects? Is Maximite finished or we should wait for the Maximite with Ethernet or WiFi or something?
There is the new Colour Maximite that will be published in the September 2012 issue of Silicon Chip. It will have eight colours on a VGA display, stereo synthesised sound and an Arduino compatible connector along with all the other features of the original Maximite (USB, BASIC language, etc). That has taken quite a lot of my time and energy to create, so after it is published I might take a break for a while before looking around for something else.
There is a couple of exclusive photos of the new colour Maximite.
Thanks, Geoff, for the interview and for Maximite. Looking forward for your further projects.
I’m personally incredibly excited that today people have started to realise that teaching computers is not only about Word and Excel, but also about understanding what is inside and how all these guts actually work. And projects like Raspberry Pi and your Maximite help enormously get beginners a bit closer to the metal.
■
// Geoff Graham, Alexander Demin
// August 2012
Palm-Size Portable Mini Travel 2 in 1 Wireless N 802.11n/g WLAN Network Router / Client Adapter
This is called WA-6220.
The official website says:
WA-6220-V1 WLAN 11n Router is designed for people on the go, user can carry it travelling around and work at anywhere. It is compliant with 802.11n specifications, up to 300Mbps data rate, provides multi-functional capabilities, particularly the high performance throughput and high-quality security. Incorporating fast Ethernet ports and is compatible to other wireless (802.11a/b/g/ n) networking device, it enables your whole network sharing a high-speed cable or DSL. Its high performance is ideal for media-centric applications like streaming video, gaming and Voice over IP technology.
It is the matchbox in size.
And can be powered from USB (I connected it directly to the laptop).
Front.
The mode switch and reset button.
The quick user guide.
Unfortunately, I haven’t met a hotel with properly installed WiFi coverage. Many can put a cable, but proper WiFi coverage with decent speed – no one. But this device makes you an ad-hoc instant router in minutes with no pain whatsoever.
P.S. By the way, a question – is WiFi hardware (chipset) in laptops, for instance, and regular WiFi access points different in any way? If the connection is duplex, it shouldn’t be. So, how to turn a laptop into a proper WiFi access point then? I tried things like Connectify on Windows 7, but it didn’t really work properly. And what should I also do on Linux and OSX?
At the time of this writing the standard Debian images (stable) distributed from the official Raspberry Pi website were built for 2GB SD cards. It is possible to copy them to larger cards and it will work but space above 2GB will be wasted.
There are a few simple steps to re-partition the card after the first boot.
Images:
http://raspberrypi.org/downloads/
Step by step:
https://projects.drogon.net/raspberry-pi/initial-setup1/
In short:
printf "d\n3\nd\n2\nn\np\n2\n157696\n\nw\n" | sudo fdisk -cu /dev/mmcblk0
sudo shutdown -r now
sudo resize2fs /dev/mmcblk0p2
http://elinux.org/R-Pi_Troubleshooting
http://elinux.org/Rpi_Low-level_peripherals
https://projects.drogon.net/raspberry-pi/wiringpi/
http://pypi.python.org/pypi/RPi.GPIO
Homepage: http://raspberry-gpio-python.googlecode.com
How to install:
cd /tmp
wget http://raspberry-gpio-python.googlecode.com/files/RPi.GPIO-0.2.0.tar.gz
tar xzvf RPi.GPIO-0.2.0.tar.gz
cd tar xzvf RPi.GPIO-0.2.0
python setup.py install
Sample program:
import RPi.GPIO as GPIO
GPIO.setmode(GPIO.BCM)
pin = 4
GPIO.setup(pin, GPIO.OUT)
while True:
GPIO.output(pin, True)
GPIO.output(pin, False)
http://codeandlife.com/2012/07/03/benchmarking-raspberry-pi-gpio-speed/
https://projects.drogon.net/raspberry-pi/gpio-examples/tux-crossing/2-two-more-leds/
http://chrishatton.org/archives/88
This kernel is based the 3.2 branch. It comes with I2C drivers and a bunch of staging drivers (for example, WiFi).
http://bootc.net/projects/raspberry-pi-kernel
http://kernel.org/doc/Documentation/i2c/dev-interface
echo ds1307 0x68 > /sys/class/i2c-adapter/i2c-0/new_device
Explanations why and how I2C was added to BootC RPi kernel:
http://bootc.net/archives/2012/05/19/i2c-and-the-raspberry-pi/
http://nathan.chantrell.net/20120519/raspberry-pi-and-the-mcp23017-i2c-io-expander/
http://robot-electronics.co.uk/files/rpi_i2c_setup.doc
Set up for using i2c with Raspberry Pi running the standard Debian squeeze distribution.
sudo dpkg-reconfigure tzdata
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install ca-certificates
Install hexxeh’s rpi-upgrade tool https://github.com/Hexxeh/rpi-update#readme:
sudo wget http://goo.gl/1BOfJ -O /usr/bin/rpi-update
sudo chmod +x /usr/bin/rpi-update
sudo apt-get install git-core
sudo rpi-update
Restart rpi for updates to finish:
sudo shutdown -r now
Download Chris Boots’s kernel with the i2c drivers in as a Debian package from his website http://bootc.net. Navigate to where the Debian package was saved using command line and install the Debian package.
dpkg -i linux-image-3.2.18-rpi1+_5_armel.deb
Copy /boot/vmlinuz-3.2.18rpi1+ to /boot/kernel.img:
sudo cp /boot/vmlinuz-3.2.18rpi+ /boot/kernel.img
Restart RPi again:
sudo shutdown -r now
Download i2c-tools:
sudo apt-get install i2c-tools
If you run ls /dev/i2c* you will not be able to see any i2c ports (e.g. /dev/i2c-0) listed. At start up the i2c ports will not be active. Make it active using modprobe:
sudo modprobe i2c-dev
Now look to see if your i2c ports exist in /dev:
ls /dev/i2c*
i2cdetect -l
dmesg | grep i2c
You should see two i2c ports listed named /dev/i2c-0 and /dev/i2c-1. /dev/i2c-0 is the one we will be using. Another way of listing them is to run i2cdetect -l (a tool that came with the i2c-tools you installed earlier).
Change permissions of the i2c-0 port to let you access:
sudo chmod 666 /dev/i2c-0
You should now be able to download and run some of our example c code. You will need to modprobe i2c-dev and change the permissions of the i2c port every time you boot up as these setting are not saved by default.
Firmware should be up-to-date. Its update is harmless and cannot brick your RPi.
/opt/vc/bin/vcgencmd version
https://github.com/Hexxeh/rpi-update
wget https://raw.github.com/Hexxeh/rpi-update/master/rpi-update -O /usr/bin/rpi-update && chmod +x /usr/bin/rpi-update
sudo apt-get install ca-certificates
rpi-update
http://iroylabs.blogspot.co.uk/2012/06/raspberrypi-mouse-and-keyboard-does-not.html
wget https://raw.github.com/Hexxeh/rpi-update/master/rpi-update
chmod +x rpi-update
mv rpi-update /usr/bin/rpi-update
sudo rpi-update
http://wrightrocket.blogspot.co.uk/2012/06/keeping-your-raspberry-pi-fresh.html
Olimex USB-SERIAL-CABLE Olinuxino console cable alternative to Raspberry Pi.
BLUE (GND) -> 0v GREEN (RX) -> GPIO 14/TxD (BCM), or 15 (wiringPi) RED (TX) -> GPIO 15/RxD (BCD), or 16 (wiringPi)
http://xbsd.nl/2011/07/pl2303-serial-usb-on-osx-lion.html
cd /tmp
wget http://xbsd.nl/~martijn/log/osx-pl2303.kext.tgz
tar xvzf osx-pl2303.kext.tgz
cd osx-pl2303.kext
sudo cp -R osx-pl2303.kext /System/Library/Extensions/
cd /System/Library/Extensions
chmod -R 755 osx-pl2303.kext
chown -R root:wheel osx-pl2303.kext
cd /System/Library/Extensions
kextload ./osx-pl2303.kext
kextcache -system-cache
Check:
kextstat -b nl.bjaelectronics.driver.PL2303
Index Refs Address Size Wired Name (Version) <Linked Against>
74 0 0xffffff7f808ee000 0xb000 0xb000 nl.bjaelectronics.driver.PL2303 (1.0.0d1) <73 34 5 4 3>
Check “kextstat” first, and if “nl.bjaelectronics.driver.PL2303” is there, it can be unloaded by:
sudo kextunload /System/Library/Extensions/osx-pl2303.kext/
http://changux.co/osx-installer-to-pl2303-serial-usb-on-osx-lio
http://planet-rcs.de/article/mac_serial_port/
Kernel 3.2.21-rpi1+ already has this driver, but it requires firmware.
Add the following line to /etc/apt/sources.list:
deb http://backports.debian.org/debian-backports squeeze-backports non-free
Update the package index:
sudo apt-get update
Install firmware-realtek deb package:
sudo apt-get install firmware-realtek
Check:
sudo iwlist wlan0 scan
Post-setup:
sudo vi /etc/network/interfaces
auto wlan0
iface wlan0 inet dhcp
wpa-ssid "your_ssid"
wpa-psk "your_password"
http://hobbytronics.co.uk/serial-vga
http://squareitround.co.uk/Resources/Punnet_net_Mk1.pdf
GMC-4 is cloned from other microcomputers, and its instruction set is compatible with:
Official magazine issues
Review and documentation
Firmware
There are no sources available of the original GMC-4 firmware.
Hardware
Blogs
Simulators
Loaders
Assemblers
Compilers
Replications
Links collection from the Official Wiki
This is a USB Bit Whacker, a development board based on PIC18. Its default firmware allows to control I/O pins via simple commands over a virtual serial port.
Schematic.
A prototype.
I was lazy to etch the board, so soldered manually. Looks unsightly but works.
In action.
After all, GMC-4 is THE thing!
P.S. The sources are available on Github – gmc4-loader. I suspect that in Python or Ruby it might be much nicer, but it was a nice exercise programming serial ports in UNIX.
Unless the multithreaded code is going to be clearly faster, then writing single-threaded code will save you a lot of headaches.
I recently purchased his book about concurrency in C++. To be honest I haven’t met better material so far about C++ 2011 memory model. Today we have an opportunity to ask Anthony a few questions, and, of course, particularly about C++.
Hi Anthony, thanks for the interview, and let me begin from afar. Programming, computers, C++… Why had you chosen this way? What dragged you on this route of bits and bytes?
I’ve always been interested in computers, and it turned out that I was reasonably good at programming them. We had a Sinclair ZX81 at home, and a BBC Micro at school when I was around 7 or 8, and it all started from there. I used to examine the source code of the games to see how they worked. When it came time to apply for jobs after my degree, I was quite clear in my mind that I wanted to do programming. It was fun, interesting, challenging, and you got paid for it!
I suspect that C++ wasn’t the first language you’ve learned. Have you ever programmed in some unusual languages at early days?
C++ wasn’t around when I started programming, so no it wasn’t the first language I learned. I started with BASIC. All home computers had some variety of BASIC built in, and it was a while before I realised there was anything else. Some programs were written in machine code, so I learnt that next — converting Z80 assembly language to hex codes by hand and typing them in as DATA statements of a BASIC loader program.
I don’t think I’ve programmed in anything that was really unusual, but I’ve used quite a few languages for various different systems. Programming a PIC with only 100 bytes of program space was a challenge, and I don’t suspect many people ever programmed the Psion Organizer II, though the programming language for it was similar to BASIC in many ways.
Can you recall any influential or maybe even outstanding books that helped establish yourself in the career?
Hmm. It would have to be the Z80 programming reference I had when I was around 10 or 11. I can’t remember the title, but I devoured that book. I knew every instruction by heart, including the hex opcodes and the clock counts. I found a similar guide to the 8086 on a shareware disk when we got a PC, printed it out and devoured that too.
These days my bookshelves have books like Design Patterns, Refactoring, and The Art of Computer Programming, though I still like to keep “close to the metal” with things like Intel’s Software Optimization Cookbook.
Now C++, and concurrency. Missing on opportunity for a few questions to the expert seems to be not such a good idea. So, to begin with, what don’t you like in the C++ 2011 concurrency support?
Good question! I don’t think there’s anything that I can point at and say “I don’t like the way that works”. I think it was a shame the “is_ready()” function got removed from future and shared_future, since it’s useful, and you can achieve the same goal with wait_for(seconds(0)) anyway.
You’re a maintainer of the Boost thread library. At the same time you develop your own library for concurrency in C++ called just::thread. What is special in this library?
Just::Thread is a strict implementation of the C++11 thread library, highly optimized for the specific platforms targetted, whereas Boost.Thread is aiming for portability, and doesn’t have the same interface or semantics in several cases. Though Boost.Thread is being brought further in line with C++11, it still lacks std::async, which is present in Just::Thread.
Just::Thread also has a special “checked” build which will identify the call chain leading to a deadlock should one occur.
Actors and the concept of sharing by communicating. It looks that the lack of this functionality doesn’t allows the concurrency in C++11 to shine in full power. Can you recommend any libraries providing such functionality?
The Just::thread Pro library currently in development will provide actors. There is also an example in my book of using message queues to create actor-style code.
I recall your saying that multi-threading in computations is tricky because it starts to make sense only after a certain volume. Do you have any recommendations on when it’s worth considering a concurrent and multi-threaded solution, and when it isn’t?
Yes, there’s an overhead to setting things up for multithreading. If your computation is quick enough then you may well be better off writing it single-threaded. Unless the multithreaded code is going to be clearly faster, then writing single-threaded code will save you a lot of headaches.
As ever, the key to optimizing for performance (and multithreading in this scenario is just that — an optimization) is to profile your application. Where is it spending its time? Can you parallelize that section? As Jason McGuiness demonstrated in his presentation at ACCU 2012, if you parallelize the wrong part of the code then it can be a lot of effort for no gain whatsoever.
Now, TDD. Where are you in this area? Have you embraced TDD and now use it everywhere? Is TDD applicable for developing concurrency libraries, for example, in C++?
I like TDD, and use it all the time. It encourages you to work in small steps, and the test suite you build up ensures that scenarios you’ve tested already don’t break as you address new scenarios.
It’s a bit harder to do TDD when developing concurrency libraries, but not impossible, and I still do it. The trick is in structuring your test to check the right thing. You typically need some form of barrier (I often use a std::promise with std::shared_future) so that you can set up all your threads in the right state, and then say “go!” The best way to test concurrent code is still to remove the concurrency — have well-defined communication mechanisms, and then test each thread in isolation.
Do you think that the code must be ideal, no trade-offs, no tolerance to inaccuracy or lack of the beauty? How do you judge your own code for being ready to release?
You should always /aim/ for perfect code. If you don’t, then it is very easy to end up with slap-dash unmaintainable code.
However, you don’t always get to write perfect code. Sometimes you can’t see a way to make it better, or the changes required are too extensive for the time frame you have. It is always better to have code that works correctly rather than code that looks pretty. If you have extensive tests then you can refactor your code later to a better structure without worrying about breaking it.
I judge code to be ready to release when I can’t think of a circumstance that would break it. If I can think of such a circumstance then I write a test for that, and fix it.
Can you name your three biggest “Never ever do that!” for programmers? and also for C++ programmers?
Tricky! “Never” is a strong word.
Here’s some things you should rarely do, in no particular order:
Use global variables. Pass things you need as parameters or hold them as member variables rather than relying on globals, as globals make access patterns hard to understand.
Use singletons. These are just global variables in disguise, and should be used with the same caution.
Write multithreaded code without carefully thinking through the data access patterns from the separate threads. If you’re going to write multithreaded code, take the time to do it properly. You’ll save yourself a lot of effort in the long run.
For C++ specifically, you should rarely:
Use malloc and free. This is C++, not C.
Write code that requires you to use “delete”. If you have to use “new” then you should probably be using a smart pointer such as std::shared_ptr or std::unique_ptr to manage the memory. Often you’re better off using a container like std::vector to manage everything anyway.
Overload operators in an unconventional manner. Sometimes it can be really useful (e.g. using << for stream insertion), but having a+b mean anything other than “adding” for the relevant data types is likely to yield confusion.
You wrote a book. Please, shed some light on this process. Why have you decided to write it in the first place, and how long did it take to prepare “C++ Concurrency in Action”? What was the most difficult part of it?
I wrote “C++ Concurrency in Action” because the opportunity came up, and I was one of those best-placed to write it, having been intimately involved in the drafting of the relevant sections of the C++11 standard.
It has taken 4 years to put it all together. One of the reasons it took so long is because the C++11 standard wasn’t finished when I started writing, and I had to add, remove and revise sections as the draft standard evolved, but even without that it was a lot of work. The hardest part was rewriting sections based on feedback from reviews.
Can you recommend other books about concurrency and multi-threading?
“Patterns for Parallel Programming” by Mattson, Sanders and Masingill is a good overview of how to structure parallel programs.
“The Art of Multiprocessor Programming” by Herlihy and Shavit is also good, but much lower level. This covers things like visibility, atomicity and consensus, and implementation of low-level structures such as queues, spin locks and monitors.
Now the question I ask everybody: sport programming and programming contests. Is it important to any developer to spend some time regularly solving puzzles? And have you even been involved in developing or implementing sophisticated algorithms?
I love puzzles. To an extent, programming is one big puzzle; that’s what makes it interesting.
I think puzzles use the same sort of thought processes as programming, so regularly doing puzzles can improve your programming. Practising anything helps make you better, so programming puzzles and contests are good for that — you have to break from your “normal” programming routine to focus yourself on something different, and try things out. I like the Intel Threading Challenge that they’ve been doing for a couple of years now. Even if you don’t enter, it can be good to do the challenges — they are fun, and really stretch you.
I did have a lot of fun once working on an algorithm for assisting helicopter maintenance engineers tune helicopters. They take a series of measurements from the helicopter with special equipment whilst it is flying in a series of different conditions, and then the software tells them what to adjust to make it fly better. Since helicopter flights are expensive, the goal was to tune the helicopter with the minimum possible set of test flights.
Are you an IDE or Vi/make guy? Do you think that it is still possible to develop sophisticated software simply in Vi, or all those days passed and considering proper IDEs is an inevitable step in enterprise software development?
I use emacs and make. I haven’t yet found an IDE that makes life easier for me. I think there’s a lot of potential in Eclipse/CDT, especially with the enhancements that Peter Sommerlad’s team are adding, but I haven’t found it enough to switch yet.
Do you think that “The Last Programming Language” exists? Or may it exist at all? Is it C++?
There is always scope for improvement in a programming language, and any given language makes some things easy to express and others hard, so I’m not sure that the “last” programming language could really exist.
If we ever get a sufficiently advanced AI, then we’ll end up using natural languages rather than specific programming languages, but I think that’s a long way off yet.
Let’s talk about hiring. If you need to hire a good C++ developer, what kind of questions you’d prefer you ask?
To some extent that depends on what I need from a new hire. If there’s time to train someone then specific C++ knowledge is not an issue, and I’d want to focus on things that highlighted general “developer aptitude”.
If I needed someone who already knew C++ then I’d want to ask questions that demonstrated that knowledge as well.
Wrapping up, do you have a list of, say, three things which each developer must implement through his or her career?
I’m not sure there are 3 things that every developer should implement. However, there are definitely things to be learnt from implementing basic data structures like lists and hash-maps, so I’ll take that as the first one.
For a second, I think a language parser can provide useful insight in putting together data structures, even if it’s a parser for a simple configuration file language rather than something complicated like C++. I think an arithmetic expression parser can be quite fun to write as a simple exercise.
Finally, I think it is worthwhile writing a client-server application. A web-app with Javascript code in the browser, and something running on the server would work for this, but anything where you are making calls from the client to the server over the network will do. The point is to do something where making a server call is considerably more expensive than making a local function call, so you have to consider the cost when designing the API. If there’s a UI then it can also be an exercise in handling backend latency whilst providing a “responsive” UI.
Thanks, Anthony. Looking forward for your new talks and books, hope for your future appearances on conferences.
■
// Anthony Williams, Alexander Demin
// July 2012
I suspect there will be series of blog posts about Raspberry Pi.
Frankly, even before receiving my own RPi I already had to solder the SD socket on another RPi which was broken. There is a rumor spreading that this type of SD sockets is very unreliable.
In contrast I cannot complain at all about an SD socket used in Maximite but it was a different type.
I decided to buy Olimex USB To Serial Cable. It was available on eBay as Olimex USB-SERIAL-CABLE Olinuxino console cable alternative to Raspberry Pi.
This cable provides 3.3V output levels which are strongly recommended for RPi (not 5V), and its 3 wires with female plugs can be directly connected to the RPi GPIO P1 header.
Unfortunately, at a time of this writing, there were no drivers for Mac OSX available on the official website, and OSX Lion (10.7.4) didn’t recognize this device. It was an unpleasant surprise because before I saw that Lion perfectly recognized other USB-RS232 hardware based on different chipsets (Microchip, for example).
I checked the Product ID and Manufacturer on the Olimex cable. When the cable is connected, click on “About This Mac > More Info… > System Report… > USB”.
Googling for a Prolific driver for Mac brought me to a great blog post, called “PL2303 Serial-USB on OSX Lion”. Following the instructions I installed the driver and connected to Raspberry Pi.
In short I did the following:
cd /tmp
wget http://xbsd.nl/~martijn/log/osx-pl2303.kext.tgz
tar xvzf http://xbsd.nl/~martijn/log/osx-pl2303.kext.tgz
cd osx-pl2303.kext
sudo cp -R osx-pl2303.kext /System/Library/Extensions/
cd /System/Library/Extensions
chmod -R 755 osx-pl2303.kext
chown -R root:wheel osx-pl2303.kext
cd /System/Library/Extensions
kextload ./osx-pl2303.kext
kextcache -system-cache
Note: Just in case, I made a copy of the “osx-pl2303.kext.tgz” file.
You can make sure that everything is installed correctly by:
kextstat -b nl.bjaelectronics.driver.PL2303
It should print the following:
Index Refs Address Size Wired Name (Version) <Linked Against>
74 0 0xffffff7f808ee000 0xb000 0xb000 nl.bjaelectronics.driver.PL2303 (1.0.0d1) <73 34 5 4 3>
Note: If you need to unload the driver for some reason, you check that the driver is loaded (by
kextstat
command above), and then:
sudo kextunload /System/Library/Extensions/osx-pl2303.kext
When the driver is installed, reconnect the cable to USB. There should be two device drivers created /dev/cu.PL2303-00002006
and /dev/tty.PL2303-00002006
. You can check it by:
ls /dev/*PL*
It should print:
/dev/cu.PL2303-00002006 /dev/tty.PL2303-00002006
Now let’s connect the cable to the RPi header P1. It’s better to disconnect it from USB temporarily.
The pinout of the cable is the following:
Wire | Desc | Rip GPIO |
Blue | GND | 0v |
Green | Rx | 14/TxD |
Red | Tx | 15/RxD |
An example of the connection:
When all three wires are attached, you can connect the cable back to USB. Now any terminal emulator software can talk to RPi via the /dev/cu.PL2303-00002006
device. I used Minicom:
minicom -D /dev/cu.PL2303-00002006 -b 115200
It should print something like this:
Welcome to minicom 2.6.1
OPTIONS:
Compiled on May 10 2012, 07:16:49.
Port /dev/cu.PL2303-00002006
Press Meta-Z for help on special keys
Debian GNU/Linux 6.0 raspberrypi ttyAMA0
raspberrypi login:
Finally, you may want to re-direct the RPi console from HDMI to the serial port (ttyAMA0). In this case RPi will be printing boot messages to the serial console instead of the monitor.
You need to login to your RPi (via SSH, telnet or just established serial connection) and execute the following:
sudo mount /dev/mmcblk0p1 /mnt
This will mount your boot partition to /mnt
. Now you need to edit the kernel commund line:
sudo vi /mnt/cmdline.txt
In cmdline.txt
make sure that console=ttyAMA0,115200
and kgdboc=ttyAMA0,115200
, and there is no another console=
assignment, for example to tty0
. After you save the changes and reboot RPi, you may see RPi boot messages in the terminal emulator connected to /dev/cu.PL2303-00002006
.
Now it is a museum of cryptography. Recently the Colossus was re-built, and now available to see it in action. My target was to see the Colossus and Enigma.
Today was a normal english day – raining all the way through. My lovely big brolly was incredibly useful.
If to skip a sticker on the windows, you may think that The War is still on.
Illustrations in the past were always created with love to details and general beauty.
A revolver. Just caught my eye because it was so close.
Frankly, I saw MP 38 (aka Schmeisser) so close first time in my life.
Do you recognized this tool? A twisted-pair crimp tool? This was from the 40th.
Looks like a key disc from Enigma-like machine.
In fact, the Enigma. To be exact – a bunch of them. I imagined the Enigma as a particular device which I saw in the American re-make of the “Das Boot” movie. But looks that even lazy people manufactured it.
A crypto device from Siemens.
Now a portable modification.
All exhibits are real and more than 70 years old.
The Bombe machine which was build particularly to crack the Enigma code.
The back view.
In action.
In a radio cabin.
On the way to The Colossus.
So, this is it – the top secret computer of the British secret service during the First World War. Recently it was re-built and now available in action.
This is a two panels stuffed with bulbs. The front panel, from the left.
The front panel, from the right.
The back view. It is all moving and cracking. The modern fans on the floor are cooling the bulbs.
The back view from the left.
Between the panels. The oscilloscope even displays some figures.
How Colossus worked.
Frankly, it wasn’t impressed at all. Yes, it is moving and cracking by tapes and spins, but… May be if they allow to code up a little program in place, I was much more excited.
In the same building with the Colossus, there is a The National Museum of Computing. Today it was, a shame, closed. Trying to bribe a person on duty to allow me to pop into mainframe and PC rooms did work out.
There is only a sad picture through the bars.
And this is the must have thing for any respectable bookstore (this is in the gift shop where they mostly sell books about the history of cryptography) – a sofa.
GMC-4 is a 4-bit microcomputer:
The architecture is described in one small document – Programming the Gakken GMC-4 Microcomputer.
Yesterday my order arrived. GMC-4 is an add-on to an issue of a Japanese magazine called “Gakken”, #24, 2009. Unfortunately it’s all in Japanese language.
Funny that the back side of the box has a table of all GMC-4 instructions.
The board, speaker, case, parts of the keyboard.
The manual in Japanese is though.
Set up batteries and the speaker.
The board.
Sticking the keyboard.
Power on… It’s alive! Alive!!!
There is the keyboard, the 7-segment LED, the processor below it, then the “hard reset” button. There are seven independent LEDS on the top. They are used, for instance, to display the current address.
Let’s write something. For example, a program waiting for a key pressed and then displaying it on the 7-segment readout.
00: 0 KA 0 ; Read a key code (0-F) into A. If pressed, then Flag=0, else Flag=1.
01: F00 JUMP 00 ; If Flag=1 (not pressed), then go to 00.
04: 1 AO ; Out A to 7-segment readout and set Flag=1
05: F00 JUMP 00 ; If Flag=1 (it is always 1 here), then jump to 00.
To enter the code: RESET 0 INCR F INCR 0 INCR 0 INCR 1 INCR F INCR 0 INCR 0 INCR RESET
To run: RESET 1 RUN
Interestingly, this device allows step-by-step trace. If to run the program via RESET 6 RUN
, it will be stopping after each instruction and displaying the current address on the top LEDs in binary code. To continue, we need to press INCR
and so on. By pressing RESET
we can interrupt the program and check out the registers (they are regular cells in memory). This is simple but convenient debugging.
Now slightly more complex example: a running light over the top LEDs displaying the number of the current one on the 7-segment readout.
00: 80 TIA 0 ; A=0
02: 1 AO ; Out A to 7-segment readout.
03: 3 CY ; Store A to Y.
04: E1 CAL SETR ; Turn on an LED which number is in Y.
06: 84 TIA 4 ; A=4
08: EC CAL TMR ; Wait (A+1)*0.1 seconds (0.5s).
0A: E2 CAL RSTR ; Turn off an LED which number is in Y.
0C: 3 CY ; Restore A from Y.
0D: 91 AIA 1 ; A = A + 1
0F: C7 CIA 7 ; If A=7, then Flag=0. Otherwise Flag=1.
11: F02 JUMP 02 ; Go to 02, if Flag=1, and set Flag=1.
13: F00 JUMP 00 ; Unconditional jump to 00 (Flag=1 after the previous instruction).
Code: 8 0 1 3 E 1 8 4 E C E 2 3 9 1 C 7 F 0 2 F 0 0
Everything is in Japanese, but Google Translate does an amazing job.
This is an amazing gadget. I started programming on Radio-86RK in machine code typing in the Monitor. Same here. Also, the issue of the magazine, which GMC-4 is bundled with, according to illustrations (unfortunately, I don’t read Japanese), is dedicated to the history of microprocessors starting from Intel 4004. Then there is a dozen of code examples and projects using GMC-4. Beautiful!
Let’s assume that:
Denote A
to be the linear machine code execution speed, B
– the speed of compiling bytecode into machine code. So, the overall execution speed is:
V(C++) = A1 + B1
V(Java) = A2 + B2
Obviously, B1 = 0
, because C++ generates native code and doesn’t require extra work at all at runtime. But B2
is always greater than zero by definition. Regardless how effective the JIT compiler is, it always requires some time to compile bytecode into native code. Furthermore, it doesn’t compile all at once. It compiles “just-in-time” when the code is actually executed. There is always a non-zero probability that the program execution flow may be paused by the JIT compiler. Even we assume that the JIT compiler uses sophisticated code prediction algorithms to reduce B2
as much as possible, it cannot eliminate B2
completely.
Now let’s have a look at A1
and A2
. These parameters define how fast the generated code is. In my personal, totally subjective and biased view, C++ (not C) has more chances for better optimization because of the templates (the compiler has full language semantics to inline) and the native machine code generation (the compiler can use the most effective machine instructions in each situation). Unfortunately, I’m not an expert in Java generics. I have just heard that they are not “native” citizens in Java. Also the Java compiler has to generate portable bytecode and cannot optimize on each particular platform. I hope, the JIT compiler tries to optimize it at runtime but there is no semantics information available anymore. Then the JIT compiler faces the trade-off between the speed and quality of compilation. C++ doesn’t have this problem because the compilation phase can be as long as necessary.
Well, I cannot convince even myself that Java can be faster than C++. I’d like to, but I have no arguments.
With Stas we did a few straightforward benchmarks based on QuickSort. Java worked ~10% slower on average.
Prior to C++ 2011 we could say that C++ had no a well-defined memory model and a standard library for concurrency. It might allow Java to beat C++ using multi-threading and concurrency. But now C++ has these features. In general either C++ or Java provide pretty low-level API for concurrency comparing, for instance, to goroutines in Go or actors in Scala (though std::async()
is quite powerful feature now in C++).
Agreed, 10% don’t always play the key role. Sometimes, advanced mechanisms of introspection, IDEs, managed execution, code hot swapping, etc. – in fact, everything what the Java platform provides, are more important than just byte threshing. But there is nothing about speed, right?
Lately I had an opportunity to play with a few devices of such sort, but slightly upper class based on ARM, not PIC32.
I apologize at once for quality of pictures, but I didn’t manage properly taking pictures of the bright computer screen neither on iPhone nor on a camera.
This is a device I have bought. It costs 70$ + delivery. There is another, more modern 1GB model already available on the official website. Mine has 512MB only.
My photos:
The USB #1. The device can be powered via this socket but they say it needs at least two amps, so powering from a regular computer may not work. I connected a keyboard here.
Mini HDMI. I also managed to connect to a DVI monitor via an adapter.
At the end face there is another USB port and an external power connector.
Micro-SD. The case of the device as a whole is a just bit bigger than a regular USB stick.
Connected to TV. Android is built-in. Booting time to this splash-screen is ~5s.
Time of the full boot is ~25s.
Connected a mouse, somehow entered a WiFi password in the virtual keyboard and checked out the proper website.
The desktop is optimized as a media center, that’s why everything is so big on the screen. After some fiddling with fonts, I had got this:
Now another proper website.
Internal storage characteristics.
The version of Android.
Pre-installed applications.
Finally, yet another proper website. Unfortunately, JSLinux wasn’t able to boot up fully. Maybe because of the web browser.
Built-in Android doesn’t require a flash card. If to prepare a card with another Linux, MK802 will boot from it. I have tried images of Ubuntu 12.04 and Lubuntu from the official website. Alas, it worked very slow and unresponsive. MK802 does require a proper distro optimized specifically for this platform.
On the official forum there are more photos of MK802. Also there are disassembly slides of a slightly different model.
Conclusion: It is not quite clear who is the audience of this device. Even Maximite allows to play with hardware, but here… Maybe running XBMC as a media center might have at least little sense.
Produced SolidRun. The device was purchased by one of my colleagues, annoyed waiting his Raspberry PI (I’m also still waiting mine), and I was able to play with it.
CPU is slower, but the peripherals are much more advanced (no WiFi, though), and they also promise 1080p video. So CuBox is more likely to be a media center.
It’s a proper cube, which side is roughly as a match box.
Power, Ethernet, HDMI, Micro-SD, USBx2, eSATA.
SPDIF, micro-USB (console).
Frankly, the device looks being made in someone’s kitchen. On the second picture it’s clearly visible that the connectors are not quite aligned with the case. But the device in general works fine. It has Ubuntu 10.04 pre-installed.
This board was given to me by another colleague of mine who had eventually got it. It came as is, without a power supply and a flash card to boot from. You have to prepare everything by yourself. Fortunately, plenty of stuff is available online.
What does make PI different from MK802 and CuBox is general purpose ports (GPIO) (in the top left corner). Theoretically you can plug in your own homemade devices into Raspberry PI. PI was developed as an educational device, not an end user one.
The market of tiny computers is exploding. They grow as mushrooms after the rain. ARM is great, apart from one little issue – Chrome isn’t yet available on it due to obvious reasons.
#include <future> #include <vector> #include <deque> #include <list> #include <forward_list> #include <typeinfo> #include <iterator> #include <iostream> template <typename T> void go(T f) { auto start = std::chrono::high_resolution_clock::now(); f(); auto stop = std::chrono::high_resolution_clock::now(); auto duration = std::chrono::duration<double>(stop - start).count(); std::cout << duration << "\n"; } void erase(std::forward_list<int>& c, std::forward_list<int>::iterator i) { c.erase_after(i); } template <typename T> void erase(T& c, typename T::iterator i) { c.erase(i); } template <typename T> void test() { std::cout << typeid(T).name() << "\n"; size_t const N = 100000; T v(N, 0); std::srand(0); for (auto t = N; t > 0; --t) { auto i = v.begin(); std::advance(i, std::rand() % t); erase(v, i); } } int main() { go (test<std::vector<int>>); go (test<std::list<int>>); go (test<std::forward_list<int>>); go (test<std::deque<int>>); }
The overloaded erase
function is for uniformity of the test
funtcion.
Compiling in VS11:
cl /O2 /EHsc test.cpp && test
Results:
class std::vector<int,class std::allocator<int> >
1.40678
class std::list<int,class std::allocator<int> >
8.85827
class std::forward_list<int,class std::allocator<int> >
8.70124
class std::deque<int,class std::allocator<int> >
9.19784
The difference isn’t huge but also not negligible at all as for list
and forward_list
.
A quote from the book:
I don’t believe anything really revolutionary has ever been invented by committee.
Huh!? Only for the sake of this quote this book is worth to be read.
Steve Jobs biography was interesting reading, even quite long though. Steve Wozniak’s book is much shorter, and also closer to people with engineering background.
Steve Wozniak tells a story how it all happened. How he spent nights at school time designing, re-designing, then again and again re-designing non-existent yet computers just on paper. It helped him develop a unique habit to always use a minimum number of chips in his circuit designs. A perfect example is a first floppy disk controller for the Apple 2 computer. Instead of a complicated and costly purely hardware solution he put a few chips and then implemented the rest in 6502 assembly language, having masterly fulfilled requirements for signal timings. Additionally, he designed, again using a few chips, a state machine, which he programmed directly in 4-bit machines codes also designed by himself (a handmade programmable logic matrix approach). Eventually, at some day Steve had invented the Apple 2, a revolutionary personal computer with the color TV-display and the full keyboard. By the way, contrary to the ordinary opinion, Jobs didn’t participate in its development. In a few years that computer made both Steves multi millionaires and started the Apple Computers. It is no wonder that the Apple surpassed everybody on the market – it was simpler, cheaper and have a bunch of unique features.
But that wasn’t the end of the story. After Steve got into a plan crash on a small airplane, operating himself, and after that incident he was suffering from partial amnesia for some time. Then he quit Apple, and, again, contrary to the ordinary opinion, he wasn’t fired, suspended or something because they all had quarreled. He just wanted to start another project – a universal remote control for consumer electronics. Formally, till now Steve is still an Apple employee, even with a salary.
But that wasn’t also the end of the story. For some reason Steve decided to organize an open air rock gig, a festival, similar to Woodstock. In fact, he did a few of them. Of course, the technology was involved. These festivals had a live satellite connection to USSR (they used equipment installed for the Olympic games in Moscow in the eightieth), right through the “iron curtain”.
After that Steve taught bases of computers at school for more than ten years and was engaged in philanthropy.
The whole slice of the book is also devoted to pranks Steve have been constantly doing through his life. For example, a Dial-A-Joke service he ran in his own apartment, or a portable TV jammer, and that big one, “The Zaltair Prank”, when Steve spread a rumor about a competing computer company, called Zaltair (remember The Altair which launched Bill Gates to the orbit?). Even Jobs believed it and only after years realised that it was a setup.
So, if you have at least a little interest in biographies, it is worth to read this engaging story about the good fellow, genius engineer, who transformed the industry of personal computers once and for all.
Seems I also have invented a pattern called “The biased attitude to your own product”. Here is a typical stream of thoughts when preparing a product release: “Oh, a little glitch here, a tiny typo there, this could be done better, that should be refactored”, etc. But when you open up a nicely packaged someone else’s product you think “Oh, man, how great and nicely polished it looks!”. But, of course, that “someone else” went through exactly the same thoughts “polishing” that product.
I’m regularly involved in producing releases, sorting bug reports, postmortem analysis, etc. I usually follow this kind of strategy: The sales grow? Yes. A number of customer issues decreases? Yes. A number of internal defects decreases? Yes. The conclusion: everything is fine, and we’re doing great. The rest is just a normal development process.
There is a “Disclaimer” link at the bottom of each post to avoid misunderstanding. I only refer to books which I read and reviewed by myself. It is not related in any way to publishers or authors.
Any information published commercially will be explicitly marked as advertising in the blog post content.
Any questions? Welcome to comments.
When you have a class where you believe you need to write a copy constructor and copy-assignment operator you are doing something wrong. // Peter Sommerlad
Today my guest is Peter Sommerlad. I was lucky to attend his sessions at the ACCU 2012 conference recently. Peter is a C++ veteran and a contributor to the C++ standard. He is a professor at HSR Rapperswil, Switzerland, and director of IFS Institute for Software (http://ifs.hsr.ch). With a team of assistants and students he develops various tools and frameworks for C++ and other languages based on Eclipse.
Now we’ll have a chance to ask Peter a few questions, which could hide behind his talks and presentations but still be very interesting for the readers.
I was very good at maths in school and in the last year I could take an additional course that included programming a programmable calculator, a HP 33 which had a stack for operands. In contrast to some of my friends who had a Casio, a TI-59 or even a Commodore V(I)C 20 home computer, I had not yet any idea of programming. However, with some explanations by our teacher I was able to “port” a TI-59 register-based lunar lander I found in a magazine to the HP 33 provided by the school. Once I got the hang of it, I “invented” a simple game: a ballistic shooting game. Goal was to hit a target at a random distance that was obstructed by a random generated wall in between. Something like Angry Birds but in in 1982 on a pocket calculator with a 10 digit seven segment LED display. You needed a lot of imagination to understand the game. You input 2 numbers: angle and velocity and the program would tell you if you hit the wall or by what distance you missed the goal. That was the first program I remember to have written myself.
That course on programming the HP 33, and the suggestion by my older sister who knew someone studying “Informatik” who earned good money while still a student made me drop my plans to sign up for a maths major and start out to study computer science (named Informatik in German) in Frankfurt/Main. Before I started university I got my first “real but home-” computer, a Sinclair ZX Spectrum, with enormous 48kBytes and cassette tape mass storage. That meant programming in BASIC where according to our maths teacher the “computer will assign the memory automatically to the named variables and you no longer needed to remember what you placed into register R1 and R2”. One of the first programs on my ZX Spectrum was drawing a red heart shape on the screen (yes, filling irregular shapes was a big problem then). I showed that my then girlfriend, but I am not sure she really appreciated that, since the computer got sometimes a bit more attention than she. I remember that, because from time to time I get reminded by her of that episode (we are now married for more than 20 years :-).
Then came university time and I had to learn assembly programming in the first semester on a Univac 1100. After the lecture on indirect addressing, one of my mates and me wrote a 2 statement program that created an endless indirect load to itself. Running it caused a “watchdog error” to occur after 2 seconds of CPU time (which was much on a mainframe then and meant all other programs were suspended for 2 seconds). After starting the program about 3 times, a guy from the operating team bursted into the computer room and shouted who was the one with our account. We slowly raised our hands and he took us to his office. After asking what we were doing and our explanation, he told us that he learned something. We had to promise never to run that program again and also never to tell that to our fellow students and got away with it. In the second semester came Pascal and now “real” programs. Pointers (in Pascal) were the first hurdle, because they were introduced without any application or motivation, such as a linked list. Only after that I was able to grok them. Some semesters later I started a job as a programmer in a small programming shop run by older fellow students. There I could actually apply my learning and got a lot of practice in solving real problems in a variety of languages, like dBaseII, Dataflex, UCSD Pascal, and Microsoft Pascal. For example, I implemented a B*-index based “data base” framework in Microsoft Pascal running on DOS.
I guess a very great impact was access to UNIX source code, especially the tools like make, awk, etc, during my diploma thesis project. I wrote a Modula-2 front end in C with lex and yacc. To be able to work at home, I had to port all the tools I needed on my 16 bit DOS computer. That required to understand the code’s inner working and structure and thus I learned from Dennis Ritchie and his colleagues a lot. It grew my taste for simple and elegant code.
One of the first books on programming that I remember was Jensen/Wirth “Pascal User Manual and Report” and then Wirth’s “Algorithms and Data Structures = Programs”. However, the latter contained many bugs when the code was blindly typed in, for example, in the version I got, merge sort was not stable, so in principle that is a stable sorting algorithm.
However, there are so many books that influenced my thinking about programming it is hard for me to pick just one. But I can pick one that I co-authored: Pattern-oriented Software Architecture: A System of Patterns aka POSA1. First, while writing it I learned so much about software, architecture and teams. As well after publishing and while presenting parts of its material on conferences I met so many influencing people who clearly shaped me heavily. Some of them I call friends today.
That is a tough question to answer. I was referring to C++11 and many language features that people learned in the past are no longer that relevant for non-library programmers. But I’d like to give you an example.
Stroustrup’s “The C++ Programming Language” introduces a class Vector that manages its memory by a pointer as one of the examples for building classes. Writing such a class in a correct, exception safe way is very hard and even C++ experts make mistakes. A reader of that book might be tempted to build his own vector or string class (women are typically not that way) which is first wrong and second completely unnecessary. If put in his first project’s code and used throughout it also introduces a maintenance burden in the long run.
It is something like the situation with Wirth’s algorithms and data structures. “All” students who had a hard time learning that will program a linked list as the first thing in their first job, because they will “need” it anyway. While that might have been true in C, we now have much better data structures in libraries that belong to our languages (not only C++). I could start ranting about the Java library’s quality, but I leave that to my colleague who is teaching algorithms and data structures in Java at our university.
Now, after we learned what not to do, I recommend to use C++’s standard library, especially the algorithms, because loops is also what you learned and what is encapsulated there. With C++11 we no longer have to do resource management by hand, but we can use std::vector, std::string, std::shared_ptr/std::unique_ptr, etc, which will do all resource management for us and allows the compiler generated default constructors and destructors to be correct automatically. When you have a class where you believe you need to write a copy constructor and copy-assignment operator you are doing something wrong. With boost this is almost also always true in C++03.
To sum up, unless you have acquired a deep understanding on library implementation practices, use libraries instead of inventing your own. Even then, avoid unnecessary DIY resource management. After myself learning more and more about how hard it is to implement the standard library components correctly and portable, I greatly appreciate those who have the experience and patience to do so. That is actually close to rocket science. But typical problems to be solved by C++ do not require that, when you use the standard library and other libraries provided by theses rocket scientists. But remember, we are on the edge of tourist space travel.
Not much, because I was late in the game. However, I managed to obtain funding from our school thanks to our headmaster Prof. Hermann Mettler to host the C++ standardization committee meeting in Rapperswil in August 2010. Some sentences in the section about std::async have been (re-)written by me with the support from many others in the committee and some other mostly editorial corrections. However, while writing my book on Simple C++ I often have to refer to the standard document and find minor mistakes regularly that will influence a future version when fixed by the editor.
How do you define hate? I can accept the standard like it is, because I know how it is actually produced. All the deficits that might annoy me are a target for fixing and working on the next version of the standard and I can work on that to do so. Sometimes you also have to accept the compromises the committee has to come up with.
If I would have been involved earlier and one of the features I championed would have been forcefully put down by my fellows I might be upset. May be, after thinking about that, I do not like that const is consistently put to the left, where logic tells me it should be put to the right. Both versions are syntactically allowed. I asked some people, if I should submit a paper to change that for a future version I was more or less forcefully held back by the elders.
First, it is header only and thus gives no hurdle such as linking to start with, especially for my students. I wrote an article on why I wrote it and it was published in an ACCU Overload magazine #75 (a slightly update version is available on http://wiki.hsr.ch/PeterSommerlad/wiki.cgi?CuTe). Main reason was that I was disappointed by CppUnit and also by another unit testing framework that I myself wrote for my work in the 1990s. Inspired by Kevlin Henney I tried to utilize the standard library and minimize the use of macros, even though, this meant, tests must be registered “manually” and aren’t “automagically” registered by static initializers (that GoogleTest will employ in its macros). I wanted to avoid relying on static initializers, because I have been bitten by them in the past with shared libraries and their inter-dependencies. The burden of test registration is mitigated by our CUTE Eclipse CDT plug-in (http://cute-test.com). The plug-in will generate the test registration code for you and even analyses if you forgot to register a test function and provides a quick fix doing so. Also nice is the diff viewer for failed ASSERT_EQUAL() checks, but that requires your values to be streamable on a std::ostream&. That provides a slight hurdle for embedded developers who want to run the testing framework on an embedded target, because of the sometimes prohibitive size of the iostream code. However, I plan to release a version of CUTE later this year that allows to configure if diff output should be provided or not, enabling a run on an embedded target that doesn’t allow iostreams.
Two of my students implemented the first TDD support for Eclipse CDT and it was productized by one of my assistants as part of our CUTE plug-in (http://cute-test.com). The TDD support will automatically generate variable definitions from using an unknown variable in code. It will furthermore generate class or enum types from using an unknown type in a declaration and it will also create (member-) function definition frames from calling an unknown function. This is very handy when either writing code in a test-driven or top-down design style. Soon the CUTE plug-in will integrate another student’s work: Mockator. Mockator generates code and build-system settings within Eclipse CDT to enable dependency injection in existing C and C++ code by so-called seams. Seams were introduced by Michael Feathers in his great book “Working Effectively with Legacy Code”. Based on these seams, Mockator can also generate test stubs and mocks for functions and types. You can read more about Mockator and its features in ACCU’s Overload magazine 108.
I know vi since about 1985. I still use it regularly to create simple files, like one-off shell scripts. I love it for being available everywhere.
However, for serious programming a good IDE is today the tool of the trade. It is like using a powerful car compared to a children tricycle to travel hundreds of miles. Yes, there are people who are fast on a tricycle, however, a car is much more comfortable. May be a car is a bad metaphor and an agile helicopter would be better, since a good IDE such as Eclipse CDT allows to quickly navigate code that it is very easy to understand an existing code base even if it is huge. Definitions and declarations are a mouse hover or a click away and the IDE usually keeps track where you come from, so you can easily get back. Any Java programmer who has used an IDE like Eclipse intensively will only consider programming in another language if similar IDE features are available. We are working hard to make Eclipse CDT at least as good for C++ than it is for Java. That is hard work and any financial sponsorship to our institute is welcome.
In our university’s program we ask our students to perform a lot of lab exercises in programming. Also at least 3-4 major one term development projects are part of the bachelor degree. Doing lab exercises in a fun way can help students learn. For example, I often give exercises in C++ to solve a problem without explicit loops or recursion, only STL algorithms are allowed. A loop solution might be obvious, but figuring out a more elegant algorithm call helps a lot.
However, some contests are set up in a way that only speed is important and not the long term effects of clean code. That is one of the reasons I either provide or ask for unit tests in my lab exercises. I even provide unit tests in exams for code to be written by the students.
Such examples for C++11 are hard to find, since compilers aren’t around so long. In addition, many current C++11 example code is used to demonstrate the new features such as move semantics without really having a use case that is benefiting from that. I’ll try in my upcoming book to give good examples. Be aware, many C++ standard library implementations are much more complex than code you want to look at, because they have to cope with strange environments (macros), provide backward compatibility and need to compensate for compiler (version) deficiencies. Solving these problems are not typical for a beginner and give a wrong impression on how to write your own code. I’ll try to lead by example, but even CUTE has some warts, because I wanted to support C++03 with boost, std::tr1 and C++11.
Software creation is art but also a craft. It requires thorough learning and practice to master. Most of art always requires also craft. They can not be separated.
I believe in code beauty. But that is relative, what is often good for beginners often suffers from the expert’s elegance. Knowing the tools available to you is essential to master them. Only knowing std::vector, for example, but not the STL’s algorithms will provide you working code, but not elegance, even if that is only counted by a function’s McCabe complexity (by the way another C++ CDT plug-in currently developed by students).
We were the first ones after Bill Opdyke to try to implement C++ refactoring. We co-created large parts of CDT’s internals to support refactoring and code generation based on ASTs (abstract syntax tree). So running the tests is not enough, code must be cleanly refactored. I do a lot of code reviews, and I often see chances to simplify it. But I doubt that talent can be automated, it is a mixture of taste and experience. It requires not only seeing the smells but also their removal. On the other hand, if you look at Martin Fowler’s smells and refactorings they often come in contrasting pairs.
Some coding standards even promote what I consider bad practice and introduce unnecessary complexity. For example, Google suggests that “out”-parameters for functions should be implemented as pointers, even in C++. The motivation behind that is, that at the call site one would need to write foo(&var) to pass the address of the variable deliberately. This is thought to improve code readability. However, while necessary in C due to the lack of references, this guideline just imposes C problems solved by C++ references onto C++ code. Within such a function, one must check that the passed in pointer is not nullptr (to use a C++11 term) and one never can be sure it is actually valid if the caller made a mistake, or one could ignore the problem at all and crash when a user passes nullptr. A C++ solution should declare the function to take a reference instead, like void foo(type & var); and an IDE will show that when you hover on the call site, without the additional need of marking it in the code by an ampersand. Using the language features effectively to write less and simpler code should be the guideline and not the protection of programmers sticking with legacy editors and not knowing their tool.
A coding standard or any rule is only good if it can actually be enforced. Tools like (PC-)lint help to do that. Often such classic tools are command-line and build-process based. More modern versions are integrated in an IDE and provide not only analysis but also quick-fixes for the found problems, for example, FindBugs for Java or FXCop for C#. Configuring them to avoid false positives without missing real problems is always an effort. To ease the application and use of lint, we created Linticator an Eclipse CDT plug-in that will automatically configure lint and run it and visualize its results within the IDE (http://linticator.com).
Being too self confident, instead do write (Unit) Tests and run them regularly. Any current software without automated tests is unprofessional in my view.
Lack of architecture or architectural knowledge is one of the big problems I often encounter when doing code reviews. This is often fostered by lack of understanding of good design.
A simple last thing: global variables also in the disguise as singletons. Pass in all dependencies from the outside as arguments. This will also show you, where you have too much coupling. Know the patterns to deal with that and writing unit tests helps you to avoid that early.
There are so many to choose from. I do not want to recommend one, because each will have its deficits. It is more important to learn more than one and those with different paradigms such as functional, object oriented, dynamically typed, statically typed, compiled, interpreted, virtual machine-based or machine language translated. An old saying is “you can write FORTRAN code in any language”. This should not be done with multiple languages, but you need to learn and apply the language’s idioms consciously.
Yes. Interaction with students is so important. It is not only them getting feedback, but it also allows me to learn from them and improve my teaching. Both kinds of feedback are important and are more efficient when immediately. A remote learning environment can not provide that bi-directional feedback, but it can be a great help for those who can not be there for some reasons. But I believe it is very expensive to provide a good remote learning environment.
I know of people to use martial arts metaphors, such as coding dojos. I have no idea about martial arts, the only sport I like, except regular cardio-vascular exercise, is skiing. This is not a team sport but allows me to get into flow and forget everything. Intensive work can be like that. But software is a team sport and regular exercises of standard moves, aka, code katas can help to shape your subconsciousness in taste for good code and style, but only if you either have that already or get the feedback to develop that. A problem I see is, that many teachers of programming novices lack the (current) practical experience of programming (including myself, but at least I try to be exposed to code regularly).
I always have asked for code examples written by the person. Once I declined a very good PhD for exactly the reason that his code was badly written and “smelly”.
Today, also the willingness to adapt to change and architectural taste is important. A candidate must understand Patterns and must know there are more than 23 GOF patterns. And better even understand that some of them like Singleton are better not employed in a software’s design.
I started out by choosing my university subject. However, the great thing about software is that we are able to invent and create our own tools. It is like if a baker would be able to create a new oven or dough mixer by baking it. So after learning a bit of it, I actually got chosen by my profession. I am doing it for about 30 years now, teaching it for about 25 years (I started teaching programming while still university student), writing articles and books about it since about 18 years and still love doing all that. I have as my goal to make this world a better world for programmers and thus hope to rid our world from bad software. I understand that I will not achieve that, but it gives me a lot of direction.
■
Thanks Peter for you time. Looking forward for your new software and talks.
// May 16, 2012
// Peter Sommerlad, Alexander Demin
#include <string> #include <cstdio> int main() { std::string s = "12345678"; std::printf("[%s]\n", s); }
This is an obvious typo of missing s.c_str()
instead of just s
. Weirdly, Visual Studio doesn’t warn you even with /Wall
. More over, the code works!. But gcc
warns though:
warning: cannot pass objects of non-POD type 'struct std::string' through '...'; call will abort at runtime
and the program dies with Illegal instruction
.
Did they really implement a trick in STL to make programs having such typo working just because this typo is very common?
So, let’s go right to the comparison table from the official web site.
I wanted to figure out unique features for the sake of what they had started this project. I checked out a presentation from OSCON 2011.
#1
#2
#3
Here is my subjective fifty cents:
There are a few principle decisions:
Personally for me, these two “features” definitely don’t fit into my way of using DVCS. For this reason I have stopped using fossil on my personal projects, though I loved the ability to have Wiki and a bug tracker built into the repository.
As a result it turns out that the key features of Veracity are “Not-GPL” and the external storages. It is obviously an attack on the corporate market. By the way, the company founder, Eric Sink, already had a DVCS product and company now acquired by Microsoft.
So, my subjective conclusion is that this is an attempt to bring DVCS into corporations.
Eric Sink wrote a book, “Version Control by Example”, in which he gives more or less fair comparison of Veracity against a few flagship DCVS. I flipped through it in half an hour, and I found a super quote, which I furiously share and actively promote (the selection in bold is mine).
11. Don’t comment out code
When using a VCS, you shouldn’t comment out a big section of code simply because you think you might need it someday. Just delete it. The previous version of the file is still in your version control history, so you can always get it back if and when you need it. This practice is particularly important for web developers, where the commented-out stuff may adversely affect your page load times.
Never use a get
prefix for getters.
Usually we consider member functions to be getters if they simply return a reference or pointer to a private member, just for the sake of data incapsulation. Such functions don’t do anything really, they have no payload.
So, there is nothing requiring a “get” verb to describe its meaning. The getter is just an alias of a member. The “get” prefix is only required when a function does a real data transformation or computation, for instance, getLastTick()
or getFullUserName()
. But it is even better here to replace “get” by something more meaningful, extractLastTick()
and buildFullUserName()
.
■
Related posts:
You can find plenty of options online to organize your standing desk. Here is my solution. It appeared by accident. When we rented an apartment, there was no room for a proper computer place in the kitchen (yes, we do need a computer place there, for sure), and my wife discovered this nice cheap table and a bar stool at IKEA.
We used it for a while as a hotspot for quick googling or ordering food online. After some time at found myself spending more and more time at this place. It was very convenient for some reason. Normally you stand of feet, and sometimes, for example, during a long “wave period”, you can place your bottom at the high chair. After we moved, I used a regular computer workplace for a while at my office at home, but then switched to the standing desk. This is how it looks:
By the way, its legs are perfect for fitting socket extensions and other stuff.
And the previous, classical workplace on the right:
P.S. By the way, I believe that people who invented a pull-out keyboard shelf for computer tables never worked at the computer. It’s hard to imagine more inconvenient way of using the keyboard. Also I personally never use any office chair if I cannot sit, resting a foot under myself.
Leonard Richardson, Sam Ruby, “RESTful Web Services”
So, skipping a tedious chapter of HTTL libraries in Ruby, Python, Java and curl, more tedious chapter on HTML4, XHTML, HTML5, Atom, XML, a little less tedious chapter about a couple of Ajax libraries, extremely tedious chapter with standard HTTP response codes, unrealistically tedious chapter about standard HTTP headers, the essence of the book can be expressed very briefly. This is my summary of the entire book.
Developing a web-service you should follow some recommendations:
http://domain/engine.php?func=123&id=test
for get user details it should be something like http://domain/users/test
.That’s it! Now you know the whole book.
This is a rare situation when I’m seriously considering money back. Unfortunately I cannot re-sell because it’s a bloody e-book.
Yesterday I was in mood and decided to install RetroBSD there.
RetroBSD is a real UNIX, derived from 2.11BSD and intended for embedded systems with fixed memory mapping. Its target platform is Microchip PIC32 microcontroller with 128KB RAM and 512KB of Flash, which can provide flexible RAM partitioning between user and kernel modes.
RetroBSD implements real preemptive multitasking and memory protection using hardware capabilities. It provides standard POSIX API (fork, exec, wait4, etc.), plus it may have a C compiler in the system allowing on-board development. The kernel is burned into the chip and the filesystem is loaded from an SD-card.
RetroBSD works not only on Maximite, but also on a few alternative boards based on PIC32 (chipKIT Max32, Sparkfun UBW32, Microchip Explorer 16, Microchip PIC32 USB/Ethernet Starter Kit, Olimex Duinomite, Duinomite-Mini and Duinomite-Mega, eflightworks).
After a little fiddling around with the bootloader and library dependencies I managed to build the binaries and upload them onto the device.
A solemn moment of powering on…
This is UNIX!
Of course, games first. The Worm.
Canfield
Now a bit of programming – Forth.
At the moment RetroBSD only communicates via the serial port and doesn’t support VGA and PS/2 interfaces provided by Maximite, but Serge Vakulenko, the author, has their support in the roadmap.
#include <future> #include <iostream> volatile int value = 0; int loop(bool inc, int limit) { std::cout << "Started " << inc << " " << limit << std::endl; for (int i = 0; i < limit; ++i) { if (inc) { ++value; } else { --value; } } return 0; } int main() { auto f = std::async(std::launch::async, std::bind(loop, true, 20000000)); loop(false, 10000000); f.wait(); std::cout << value << std::endl; }
Compiling via clang:
clang++ -std=c++11 -stdlib=libc++ -O3 -o test test.cpp && time ./test
Run:
SSttaarrtteedd 10 2100000000000000
11177087
real 0m0.070s
user 0m0.089s
sys 0m0.002s
Obviously, the increment and decrement operations aren’t atomic, and the value
variable contains garbage at the end.
#include <future> #include <iostream> volatile int value = 0; int loop(bool inc, int limit) { std::cout << "Started " << inc << " " << limit << std::endl; for (int i = 0; i < limit; ++i) { if (inc) { asm("LOCK"); ++value; } else { asm("LOCK"); --value; } } return 0; } int main() { auto f = std::async(std::launch::async, std::bind(loop, true, 20000000)); loop(false, 10000000); f.wait(); std::cout << value << std::endl; }
Run:
SSttaarrtteedd 10 2000000100000000
10000000
real 0m0.481s
user 0m0.779s
sys 0m0.005s
Now value
has the correct value at the end, but, of course, this is a dirty non-portable hack for x86 only. For example, this code works for me only if compiled with -O3
. Otherwise it crashes with “illegal instruction” because the compiler injects extra stuff between the LOCK instruction and the following increment or decrement.
#include <future> #include <iostream> #include "boost/interprocess/detail/atomic.hpp" using namespace boost::interprocess::ipcdetail; volatile boost::uint32_t value = 0; int loop(bool inc, int limit) { std::cout << "Started " << inc << " " << limit << std::endl; for (int i = 0; i < limit; ++i) { if (inc) { atomic_inc32(&value); } else { atomic_dec32(&value); } } return 0; } int main() { auto f = std::async(std::launch::async, std::bind(loop, true, 20000000)); loop(false, 10000000); f.wait(); std::cout << atomic_read32(&value) << std::endl; }
Run:
SSttaarrtteedd 10 2100000000000000
10000000
real 0m0.457s
user 0m0.734s
sys 0m0.004s
The result is correct, and the time is almost the same as with LOCK
. Not surprisingly, atomic
under cover also uses LOCK
but in a portable and guaranteed way.
#include <future> #include <iostream> #include "boost/smart_ptr/detail/spinlock.hpp" boost::detail::spinlock lock; volatile int value = 0; int loop(bool inc, int limit) { std::cout << "Started " << inc << " " << limit << std::endl; for (int i = 0; i < limit; ++i) { std::lock_guard<boost::detail::spinlock> guard(lock); if (inc) { ++value; } else { --value; } } return 0; } int main() { auto f = std::async(std::launch::async, std::bind(loop, true, 20000000)); loop(false, 10000000); f.wait(); std::cout << value << std::endl; }
Run:
SSttaarrtteedd 10 2100000000000000
10000000
real 0m0.541s
user 0m0.675s
sys 0m0.089s
A bit slower but not much.
#include <future> #include <iostream> std::mutex mutex; volatile int value = 0; int loop(bool inc, int limit) { std::cout << "Started " << inc << " " << limit << std::endl; for (int i = 0; i < limit; ++i) { std::lock_guard<std::mutex> guard(mutex); if (inc) { ++value; } else { --value; } } return 0; } int main() { auto f = std::async(std::launch::async, std::bind(loop, true, 20000000)); loop(false, 10000000); f.wait(); std::cout << value << std::endl; }
Run:
SSttaarrtteedd 10 2010000000000000
10000000
real 0m25.229s
user 0m7.011s
sys 0m22.667s
Now it works much slower, really.
Method | Time (sec.) |
No synchronization | 0.070 |
LOCK | 0.481 |
Atomic | 0.457 |
Spinlock | 0.541 |
Mutex | 22.667 |
Of course, the result depends really on the platform and the compiler (I tested on Mac Air and clang). But for me it was quite interesting to see that spinlock, in spite of its more sophisticated implementation comparing to atomics, works not much slower.
Sadly, my clang 3.1 still doesn’t support atomic
, and I had to use boost.
T*
or T&
(#2). Therefore, when I wanted Pimpl, I didn’t use smart pointers because it seemed that it needed the full class definition.
A.h
:
#include <memory> class A_pimpl; class A { A(); ~A(); std::unique_ptr<A_pimpl> p; }
I believed to some reason that it will not work because the class A_pimpl
is partially defined. I was surprised as a child when tried and realised that it works! The fact #1 is not equivalent to the fact #2.
Now in A.cpp
I can write:
#include "module.h" #include "pimpl.h" A::A() : p(new A_pimpl()) {} A::~A() {}
Everything above also works for std::shared_ptr
(C++ 2011), boost::scoped_ptr
, and boost::shared_ptr
.
Update
It is important that class A
must an explicitly provided destructor. Moreover, its body must in A.cpp
, not in the header. Otherwise, it will be a compilation error, for example, “error C2338: can’t delete an incomplete type”.
His presentation is available with examples.
Unfortunately, I missed his second presentation, “Dataflow, Actors and High Level Structures in Concurrent Applications”. In fact, it was about implementing the concept of actors, similar as in Erlang, in C++ 2011.
This presentation is also available with examples.
Recommend.
Going further. Anthony’s book, “C++ Concurrency in Action”, has been released recently.
If you want an exhaustive explanation about the C++ memory model, please read the chapters 5 (along with all others).
#include <future> int main(int argc, char* argv[]) { for (auto i = 0L; i < 1000000; ++i) { auto f = std::async([](){ return 0; }); f.get(); } return 0; }
This code consistently crashes (and doesn’t throw). When decreasing a number of iteration, at some point to stops crashing.
The compiler cl.exe 17.00.40825.2, IDE 11.0.40825.2 PREREL. Posted on Stack Overflow. They say, likely it is the bug.
Where to file a bug to VS?
This is my naive approach:
int naive_quick_sort(std::vector<Type>::iterator begin, std::vector<Type>::iterator end) { auto const sz = end - begin; if (sz <= 1) return 0; auto pivot = begin + sz/2; auto const pivot_v = *pivot; std::swap(*pivot, *(end - 1)); auto p = std::partition(begin, end, [&](const Type& a) { return a < pivot_v; } ); std::swap(*p, *(end - 1)); if (sz > 4096) { auto left = std::async(std::launch::async, [&]() { return naive_quick_sort(begin, p); }); naive_quick_sort(p + 1, end); } else { naive_quick_sort(begin, p); naive_quick_sort(p + 1, end); } return 0; } void quick_sort(std::vector<Type>& arr) { naive_quick_sort(arr.begin(), arr.end()); }
Everything is simple, but it’s worth to consider some details. This is a threshold, 4096
, determining when we switch the parallelism off. Why 4096
? Just a guess, without a particular explanation.
The are three candidates:
async
)if (sz > 4096)
to if (false)
)naive_quick_sort(arr.begin(), arr.end())
to std::sort(arr.begin(), arr.end())
)We sort an array of 50000000 elements of the type int64
(signed) with 10 runs and take an average. The values are random:
std::tr1::uniform_int<Type> uniform( std::numeric_limits<Type>::min(), std::numeric_limits<Type>::max()); std::mt19937_64 engine; void generate(std::vector<Type>& v) { std::for_each(v.begin(), v.end(), [](Type& i) { i = uniform(engine); }); }
Don’t ask me, why I convert the data between big and little endians back and fourth. It was implemented to compare this implementation with another one, in Java. In measurements we count only the “pure” time.
The compiler is VS 2011, 64-bit. CPU Intel Core i5 2.53GHz, 4 cores.
Iteration With async() One thread std::sort()
--------- --------------- ------------ ------------
1 2512 6555 7309
2 2337 6320 6977
3 2450 6516 7180
4 2372 6388 6933
5 2387 7074 7189
6 2339 7399 7040
7 2434 6875 7040
8 2562 7060 7187
9 2470 7050 7145
10 2422 6846 6898
--------- --------------- ------------ ------------
Среднее 2428.5 6808.3 7089.8
The time is in milliseconds.
It turns out that the naive implementation based on async()
is three times faster then the single threaded std::sort()
. A weird slowdown of the std::sort()
against my naive single thread version can be explained, perhaps, that the data I generated is good, and my naive implementation is just lucky.
Any practical use out of it at all? Probably not. It is difficult to predict the behaviour of the implementation on different sets of data. For example, how to choose the threshold properly? Is it worth to start using thread pools?
The full source is below, including the generator.
Build and generate the data:
call "%VS110COMNTOOLS%..\..\VC\vcvarsall.bat" amd64 && ^
cl /Ox /DWIN32 sort_async.cpp && ^
sort_async generate
Warning! It generates 8GB of data.
Build and test:
call "%VS110COMNTOOLS%..\..\VC\vcvarsall.bat" amd64 && ^
cl /Ox /DWIN32 sort_async.cpp && ^
sort_async
File sort_async.cpp
:
#include <vector> #include <iostream> #include <fstream> #include <sstream> #include <algorithm> #include <iomanip> #include <future> #include <random> #include <chrono> #include <cstdlib> const int ITERATIONS_NUM = 10; const int DATA_SIZE = 50000000; typedef __int64 Type; inline void endian_swap(Type& x) { x = (0x00000000000000FF & (x >> 56)) | (0x000000000000FF00 & (x >> 40)) | (0x0000000000FF0000 & (x >> 24)) | (0x00000000FF000000 & (x >> 8)) | (0x000000FF00000000 & (x << 8)) | (0x0000FF0000000000 & (x << 24)) | (0x00FF000000000000 & (x << 40)) | (0xFF00000000000000 & (x << 56)); } std::tr1::uniform_int<Type> uniform( std::numeric_limits<Type>::min(), std::numeric_limits<Type>::max()); std::mt19937_64 engine; void generate(std::vector<Type>& v) { std::for_each(v.begin(), v.end(), [](Type& i) { i = uniform(engine); }); } void check_sorted(const std::vector<Type>& v, const std::string& msg) { for (auto i = 0; i < v.size() - 1; ++i) { if (v[i] > v[i + 1]) { std::cout << "\nUnsorted: " << msg << "\n"; std::cout << "\n" << i << "\n"; std::cout << v[i] << " " << v[i + 1] << "\n"; std::exit(1); } } } std::string data_file_name(const int i, const std::string& suffix) { std::ostringstream fmt; fmt << "trash_for_sort_" << i << suffix << ".bin"; return fmt.str(); } void save_file(std::vector<Type> array, const std::string& name) { std::for_each(array.begin(), array.end(), [](Type& i) { endian_swap(i); }); std::ofstream os(name.c_str(), std::ios::binary|std::ios::out); auto const bytes_to_write = array.size() * sizeof(array[0]); std::cout << "Saving " << array.size() << " bytes to " << name << "\n"; os.write((char *)&array[0], bytes_to_write); } int main_generate(int argc, char* argv[]) { std::cout << "Generation\n"; for (auto i = 0; i < ITERATIONS_NUM; ++i) { std::vector<Type> unsorted(DATA_SIZE); generate(unsorted); save_file(unsorted, data_file_name(i, "")); std::cout << "Sorting...\n"; std::sort(unsorted.begin(), unsorted.end()); check_sorted(unsorted, "check sorted array"); save_file(unsorted, data_file_name(i, "_sorted")); } return 0; } void load_file(std::vector<Type>& array, const std::string& name) { std::cout << "Loading " << name; array.resize(DATA_SIZE, 0); std::ifstream is(name.c_str(), std::ios::binary|std::ios::in); auto const to_load = array.size() * sizeof(array[0]); is.read((char *)&array[0], to_load); if (is.gcount() != to_load) { std::cerr << ", Bad file " << name << ", loaded " << is.gcount() << " words but should be " << to_load << "\n"; std::exit(1); } std::for_each(array.begin(), array.end(), [](Type& v){ endian_swap(v); }); } int naive_quick_sort(std::vector<Type>::iterator begin, std::vector<Type>::iterator end) { auto const sz = end - begin; if (sz <= 1) return 0; auto pivot = begin + sz/2; auto const pivot_v = *pivot; std::swap(*pivot, *(end - 1)); auto p = std::partition(begin, end, [&](const Type& a) { return a < pivot_v; } ); std::swap(*p, *(end - 1)); if (sz > 4096) { auto left = std::async(std::launch::async, [&]() { return naive_quick_sort(begin, p); }); naive_quick_sort(p + 1, end); } else { naive_quick_sort(begin, p); naive_quick_sort(p + 1, end); } return 0; } void quick_sort(std::vector<Type>& arr) { naive_quick_sort(arr.begin(), arr.end()); } int main(int argc, char* argv[]) { if (argc == 2 && !std::strcmp(argv[1], "generate")) return main_generate(argc, argv); std::vector<double> times; auto times_sum = 0.0; for (auto i = 0; i < ITERATIONS_NUM; ++i) { std::vector<Type> unsorted; load_file(unsorted, data_file_name(i, "")); std::vector<Type> verify; std::cout << ", "; load_file(verify, data_file_name(i, "_sorted")); check_sorted(verify, "verify array"); std::cout << ", Started"; auto start = std::chrono::high_resolution_clock::now(); quick_sort(unsorted); auto stop = std::chrono::high_resolution_clock::now(); std::cout << ", Stopped, "; auto duration = std::chrono::duration<double>(stop - start).count(); std::cout << duration; check_sorted(unsorted, "sorted array"); const auto match = unsorted == verify; std::cout << (match ? ", OK" : ", DON'T MATCH"); times.push_back(duration); times_sum += duration; std::cout << "\n"; } auto const average = times_sum / ITERATIONS_NUM; auto const max_element = *std::max_element(times.begin(), times.end()); auto const min_element = *std::min_element(times.begin(), times.end()); auto const average_fixed = (times_sum - max_element - min_element) / (ITERATIONS_NUM - 2); std::cout << "Average: " << average << "s, " << "Average without max/min: " << average_fixed << "s." << std::endl; }
Finally, a CPU utilization graph. We see 100% spikes on each iteration.
There is an article from Microsort, “Dynamic Task Parallelism”, also showing how to implement multithreaded Quicksort.
YYYY
, MM
, mm:ss
, etc). Developing the static blog engine for this blog in Go, I’ve come across another approach – a use the real hardcoded values instead of the special symbols. For instance:
func format_time(t time.Time) string { return t.Format("2006.01.02-15.04.05") // Similar to YYYY.MM.DD-hh.mm.ss }
A full list of “magic” values (from time/format.go):
stdLongMonth = "January" stdMonth = "Jan" stdNumMonth = "1" stdZeroMonth = "01" stdLongWeekDay = "Monday" stdWeekDay = "Mon" stdDay = "2" stdUnderDay = "_2" stdZeroDay = "02" stdHour = "15" stdHour12 = "3" stdZeroHour12 = "03" stdMinute = "4" stdZeroMinute = "04" stdSecond = "5" stdZeroSecond = "05" stdLongYear = "2006" stdYear = "06" stdPM = "PM" stdpm = "pm" stdTZ = "MST" stdISO8601TZ = "Z0700" // prints Z for UTC stdISO8601ColonTZ = "Z07:00" // prints Z for UTC stdNumTZ = "-0700" // always numeric stdNumShortTZ = "-07" // always numeric stdNumColonTZ = "-07:00" // always numeric
A few examples:
ANSIC = "Mon Jan _2 15:04:05 2006" UnixDate = "Mon Jan _2 15:04:05 MST 2006" RubyDate = "Mon Jan 02 15:04:05 -0700 2006" RFC822 = "02 Jan 06 15:04 MST" RFC822Z = "02 Jan 06 15:04 -0700" // RFC822 with numeric zone RFC850 = "Monday, 02-Jan-06 15:04:05 MST" RFC1123 = "Mon, 02 Jan 2006 15:04:05 MST" RFC1123Z = "Mon, 02 Jan 2006 15:04:05 -0700" // RFC1123 with numeric zone RFC3339 = "2006-01-02T15:04:05Z07:00" RFC3339Nano = "2006-01-02T15:04:05.999999999Z07:00" Kitchen = "3:04PM" // Handy time stamps. Stamp = "Jan _2 15:04:05" StampMilli = "Jan _2 15:04:05.000" StampMicro = "Jan _2 15:04:05.000000" StampNano = "Jan _2 15:04:05.000000000"
I didn’t see such approach before, I’ve found it clearer and more logical.
std::endl
is always better that \n
in C++ streams (portability, for instance). It turns out that no.
A code with std::endl
:
#include <string> #include <iostream> int main() { for (int i = 0; i < 1000000; ++i) { std::string s(1, 'x'); std::cout << s << std::endl; } return 0; }
Compile and run:
clang++ -o endl -O3 endl.cpp && time ./endl >rubbish
real 0m4.518s
user 0m1.080s
sys 0m3.311s
A code with \n
:
#include <string> #include <iostream> int main() { for (int i = 0; i < 1000000; ++i) { std::string s(1, 'x'); std::cout << s << '\n'; } return 0; }
Compile and run:
clang++ -o endl -O3 endl.cpp && time ./endl >rubbish
real 0m0.263s
user 0m0.236s
sys 0m0.008s
The difference is obvious. The second one is much faster.
std::endl
always flush
es the stream. In turn, \n
simply puts a new line character to the stream, and in most cases this is exactly what we need. And std::flush
, if necessary, can be called afterwards, once, explicitly.
Yesterday the book had arrived. I wanted to write a proper detailed review, but realised it doesn’t make sense at all. This is a must have book if you’re interested in the new C++ 2011. The book is an exhaustive reference with a lot of examples of the new STL and the language itself. In my view, this is the first big C++ 2011 book at the moment.
It is called live coding. What You Code Is What You See - WYCIWYS.
I thought his demos were hardcoded. But it turns out, there is a real programming language allowing such crazy things in runtime. Check out a short video:
The language is called circa. Currently it is alpha. On my amateurish opinion in the game development, it is very convenient for experiements with gameplay, at least for 2D games.
Here is an interesting article from the author about preserving the runtime state on live code changes.
A collection is videos with more live coding.
assert
macro is enough for unit testing.
For instance, a mini project having one file, and you don’t want to drag the Google Test or cmockery in. I usually do something like this:
#include <cassert> void foo(...) { // something } ... #ifdef UNIT_TESTING void Test_for_a_particular_use_case() { // Initialization ... assert(condition_1); ... assert(condition_N); } ... int main(...) { Test_for_a_particular_use_case(); std::cout << “All tests passed.” << std::endl; return 0; } #else int main(...) { // a proper main } #endif
Of course, you still can forget to add your test to main()
, and it seems working, but the test simply doesn’t run.
But! All this is trifles in comparison with the advantages which the tests provide.
By the way, I’ve noticed an interesting habit: when I’m writing a function working with files, I always create two:
void FunctionDoingSomethingFromStream(std::istream* is) { ... } void FunctionDoingSomethingFromFile(const std::string& filename) { std::ifstream is(filename); return FunctionDoingSomethingFromStream(&is); }
The first function is perfect for testing because you can pass in a mocked std::istringstream
. The second doesn’t really need to be tested due to its simplicity. It may be tested though in QA, but not in unit testing during within build.
There is another interesting trick from Kevlin Henny. Usually a subject of testing (a class or a function) has a pre-condition and a post-condition. It’s worth to highlight these stages in comments by “Given”, “When” and “Then” words. For example:
void Test_for_a_particular_use_case_to_check() { // Given: ClassToTest a; // When: a.do_this(...); a.do_that(...); a.setup_something(...); // Then: assert(condition_1); ... assert(condition_N); }
Such explicit split enforces proper test structuring: without loop and branches, and testing for only one scenario (for another scenario it will be another test).
■
For example, my PS1
variable is:
\W `vcprompt -f "%m%u %s:%b"`\$
making the prompt in bash
to be like this:
_engine +? git:master$
It says that the current directory is _engine
and there is a git repository here, the current branch is master
, and there are changed (+
) and also untracked files (?
). If there is no repo in the current repository, the prompt will be normal.
vcprompt supports not only git, but bzr, cvs, darcs, fossil, hg, svn.
It requires Python, and is also hardly usable on Windows.
Because of laziness I began using Blogspot. It gives plenty of bells and whistles: templates, widgets, instant indexing by Google, statistics, even comments become to be a tree at some point. Everything seems to be great, but alas, the online Blogspot editor isn’t designed to create programmer posts. When it requires to insert a code or a table, sufferings begin. For example, for my another blog, not about programming, called “Boiled eggs, sir!”, “the capabilities” of Blogspot are more or less enough.
I want to keep sources of the posts in a nice and clear format, not spoiled by HTML. It turned out that the posts sources are spread across my computer here and there, sometimes in multiple copies. You begin creating a post simply in an editor only formatting paragraphs, without links and images, and eventually you save the final version. Then you compose HTML during that the post body maybe also changed. Thus I’m lazy to update the original, non-HTML version, so finally, only the spoiled by HTML version remains up to date. But this is not the end of the store still. Often after publishing you spot on a typo, go to the Blogspot online editor and fix it right on the page. Again, that very first document and its local, prior to uploading the HTML version, don’t have all the corrections. Ultimately, only the online edition in Blogspot is relevant. Of course, it is possible to export the entire blog from Blogspot (maybe even automatically, on regular basis) and back them up, but again, everything will be in the HTML format.
Some time ago I began using ReST. The life had become a bit easier. ReST allows to write text in the more or less predictable layout (paragraphs, links, code), and then HTML is generated from it, and then later pasted (again, manually) into the Blogspot editor. All my attempts using Google Command Line, in fact, failed. The problem of the outdated original documents (after online fixed) still stood. Furthermore, ReST didn’t solve the problem with pictures. I had to upload them somewhere in advance to do the a proper local preview.
I almost cannot explain why, but the idea of dynamic blog engines like WordPress scared me. Use of a database to keep posts seemed to me unnecessarily complicated.
I was almost ready to choose an intermediate solution – Doku Wiki, for example, vak.ru. This engine is dynamic, but it stores the posts as files. Plus, it supports versioning. Doku can be used as the engine for the whole site, not only the blog. Though its design is unsightly, but pictures and other arbitrary attachments are handled by Doku automatically.
There was another option, which I almost signed up for – TiddlyWiki. TiddlyWiki is my favorite tool on Windows for notes. Why only Windows? Because on Mac I simply use regular plain text files in Documents or Desktop, and Spotlight provides the instant search. Therefore on Mac the instant search, the killer feature of TiddlyWiki, didn’t make any sense. But I digressed.
It turns out, there are TiddlyWiki fans which converted it to a blog engine, a static-dynamic mutant.
For instance, here is an example of a blog running TiddlyWiki – Rich Signell’s Work Log. Esoterics, in my opinion. It isn’t quite clear even how to implement comments, at least Disqus. But surprisingly, there is public hosting based on Tiddly – tiddlyspot.
Finally, I had got hooked on purely static blog engines. The charm of it is that such blog can be hosted anywhere. Neither database nor even no server scripting is required. But further – it is more. GitHub and Heroku provide not only free hosting but also git as the CMS.
For example, there is a static blog engine called Jekyll. In Jekyll posts are created in Markdown or Textile, therefore the formatting issue is solved. No ugly HTML anymore. In fact, the engine can drive the entire website, not only the blog, plus some source files are deployed as a blog.
The comments, as the main dynamics in blogs, can be implemented via, for instance, Disqus. By the way, there is a quite popular pure zen approach to the comments – static ones (to me, such word combination is the oxymoron in the first place). With the static comments there is a comment entering form somewhere on the page along with the statically rendered existing comments. You enter a comment and submit. Then it gets sent to the blog owner. He or she approves (or rejects) it, clicks somewhere, and the comment is placed to a file. Then the blog is re-compiled and finally deployed to public. Obviously, it is far from real-time, which is the heart of commenting, in my humble opinion.
I very much appreciate discussion, and such approach is not for me. I use Disqus. By the way, all the comments can be easily exported from Disqus, and, for instance, converted to a static form.
But let’s go back to Jekyll. GitHub directly supports it (its author is the GitHub co-founder) and can render Jekyll projects for you (of course, you still can render locally). You simply git push
your stuff to GitHub, and after a moment it becomes visible in GitHub Pages.
Heroku works slightly different. Heroku hosts Ruby, therefore a Heroku project is a bunch of pages, plus a mini web server application serving them. Looks a bit scary, but such server in Ruby is very simple:
require 'bundler/setup' require 'sinatra/base' class SinatraStaticServer < Sinatra::Base get(/.+/) do send_sinatra_file(request.path) {404} end def send_sinatra_file(path, &missing_file_block) file_path = File.join(File.dirname(__FILE__), 'public', path) file_path = File.join(file_path, 'index.html') unless file_path =~ /\.[a-z]+$/i File.exist?(file_path) ? send_file(file_path) : missing_file_block.call end end run SinatraStaticServer
As strangely enough it sounds, Heroku as blog hosting is simpler than GitHub. Additionally, your git repository on Heroku remains private, whereas on GitHub by default it is public. I don’t see any problems with it though, because it is all available anyway to everybody via the web site, but some people prefer keeping blog guts or bits of unfinished work in private.
Also, both GitHub Pages and Heroku support a proper second level domain if you have one.
Well, I had chosen Jekyll and Heroku hosting. Alas, vanilla Jekyll has no HTML templates or themes, and you have to design it. If you’re lazy, you can try Octopress.
Octopress is a static blog engine based on Jekyll. “Based” means, this is Jekyll plus a bunch of useful plugins, a very nice HTML5-compatible theme, and scripts helping in deployment to GitHub Pages and Heroku.
So, I had installed Octopress, tried a couple of posts, rendered locally, deployed to GutHub and Heroky. Everything was amazing, I had began the most tedious part – converting posts from lovely Blogspot. In fact, it was almost pure manual cut-and-paste work. Three weeks of struggle, and my poor three hundred posts were done.
Everything was ready to launch my nice or shiny static blog. But here I had met my main disappointment. The precious Jekyll, written in Ruby, rendered my hundred posts (attention!) 15 minutes (on Mac Air). As you can understand, at the beginning I had to rebuild a hundred of times – design changes, formatting etc. So, 15 minutes wasn’t even close to any acceptable time, at all.
At random, I found a bottle neck in Jekyll/Octopress – the majority of those 15 minutes were spent generating the RSS file, atom.xml
. For some reason, in original templates, that RSS filed included only the latest twenty posts. But my blog is small, and I included all posts there. It had caused that 15 minutes build.
All this seemed to me an absurdity (at all my love to Ruby). After some analysis (at that time I began to understand how Jekyll worked, more or less) and unwillingness to fix Jekyll trying to speed it up, I asked a question – what if I develop the static engine by myself? using Jekyll’s principles? After all it is only work with files, strings, and templates. Besides, I wanted to blog in two languages, but Jekyll didn’t support that. Having my own engine seemed to be a very cool idea because I would free to implement exactly that I wanted.
Instrumentation? As a real man in C++/boost? Perhaps. I suspect it would work very fast, but it is boring. I’d chosen Go. It is natively compiled, has simplified memory management (thanks to the garbage collector), regular expressions, maps, templates, Markdown library. Everything, except the later, is out of the box. There should not be any performance issues. Here just the first release of Go had been arrived, now installation on both Windows and Mac became much easier.
So, after three nights, my own wheel had been made – Goblog. The project is fully open. The web site and its source are in the same repository.
There are two main locations: a project and a target web site/blog. The first contains source files. In the process of assembly the project files are copied to the target directory preserving the relative directory structure. By default, the files are copied “as is”, as binaries. If a file has one of the following extensions: html
, xml
, or js
, it is processed by Go template library. Files with the markdown
extension are also processed by the Markdown library.
Directories:
<root>
– Here is the compiled/built web site as it is visible at http://demin.ws/.<root>/_engine
– This is the project: source files and the blog generator.Subdirectories and files in _engine
:
_includes
– Include files available by {{include “filename”}}
macro.
_layouts
– Layouts (see below).
_site
– Actually, the source directories and files. This directory is the root of a future site. Files from here are copied to the target directory. Some files are processed by templates.
_posts
– Posts. These files are process specially. Besides templates, they are renamed according to the blog structure, where the date is included to URL: domain/blog/year/month/day/post-title
.
The posts are Markdown files having a special preamble and name. These files are deployed to a separate directory, /blog
. The post meta information is collected and propagated to the template place holders. Also the post sources are used to build the reverted index of search.
The concept of layouts is grabbed from Jekyll. If a post or a page has the layout
attribute in its preamble (for example), a layout of the given name is loaded (from _layouts
directory), the post body is put to the Page.child
place holder, the layout is rendered, and the result becomes the final HTML page of the post. This is incredibly useful for similar pages, like posts. The layouts can be nested.
Now, the generator – main.go.
To build the blog, I go to the _engine
directory and do:
make
It prints roughly the following:
_engine$ make
gofmt -w=true -tabs=false -tabwidth=2 main.go
go run main.go
Go static blog generator Copyright (C) 2012 by Alexander Demin
Words in russian index: 18452
Words in english index: 3563
15.672979s
Processed 344 posts.
If no errors, in the root directory (in ..
with regard to _engine
) the compiled files should be created, ready for deployment. On my Mac Air the build takes 15 seconds (hello, Jekyll/Octopress and goodbye). The entire project is under git, so it is clear anytime which files are new, deleted, or changed.
Then the built site can be viewed locally (see below).
When you are happy with changes, let’s add and submit them (the sources from _site/
along with the assembled files) locally:
git add ../*
git commit -m "New post about ..."
At last, let’s deploy to GitHub:
git push
Almost immediately the files become visible at demin.ws.
In Makefile there is a couple of extra commands making life easier.
To check/preview the site locally I temporarily add 127.0.0.1 demin.ws
to /etc/hosts
and launch a mini web server. Remember, how it looked in Ruby? Tiny, right? Check out this:
package main import "net/http" func main() { panic(http.ListenAndServe(":80", http.FileServer(http.Dir("..")))) }
Nice? Now run it:
go run server.go&
It may require sudo
to acquire the port 80.
Generally speaking, there is no need touching /etc/hosts
, just use localhost:80
. Only for checking the RSS file, atom.xml
, having full URLs, you have to fiddle with hosts.
As a Markdown extension I have a special tag for blocks of code:
{% codeblock lang:xxx %}
...
{% endcodeblock %}
I inherited the idea from Octopress. Markdown library already has the syntax for code:
``` xxx
...
```
where xxx
is a language.
My own tag aims to add additional attributes easier, for example, turning on the line number, tabs conversion, etc.
Then I had to decide how to highlight the syntax. I tried a couple of online libraries, which color right on the page, but all of them had minor glitches here and there, and I went for static coloring.
The first, which came to my mind, was pygments. It seemed okay, but thanks to the Python, it worked really slow. The total built time had increased from 15 seconds to two minutes. The main time was spent for a code coloring. I started thinking about a cache for the already highlighted code and similar nonsense, but after a little research the problem was solved radically.
I just needed a colorizer, implemented in a language suitable for this problem. I found two options: GNU Source-highlight and Highlight. Both are in C++, the both worked almost instantly.
For example, a guy compared pygments vs syntax-highlight.
I liked Highlight because it supports more languages (for example, the first one even didn’t support Go). After migration to Highlight, the built time came to normal, ~15-16 seconds, and I was satisfied.
The colorizer is invoked from the regular expression callback parsing the {% codeblock %}
tag (function highlight()).
There are plenty of editors having Markdown preview. I use MarkdownPad for Windows and Marked on Mac.
At the moment I have decided not to use tags at all. Based on a personal experience, I have realised that I never use tag neither in my blog nor in others. Besides, views on the post categorization changes over time, and at times you have to maintain a completely outdated tag just for the sake of uniformity. For example, what is the point of the c++
tag in my blog? Who on Earth even used it?
But the minimalism isn’t a way to make life more complicated. On the contrary, I search in my old posts all the time. With Blogspot I went to the front page and used ⌘-F (sorry, CTRL-F) to search across the post titles. Exactly for this purpose I started placing almost all informative posts into the right column on the front page.
In the new blog everything “works” similar with the post catalog on the front page. During the migration I had improved some titles to be more informative for search.
But! All of this doen’t matter anymore because of the fully functional context search capability in the new blog.
One of annoying inconveniences of Jekyll is that it never checks anything in the content. But I passed though it fully converting posts from Blogspot: broken links, missing quotes, bad dates and so on. That’s why the Goblog checks everything – formats, links, overall semantics, and it stops the build in the case of an error. When I added the check_links() function, I had found a bunch of dead links.
There was another issue with Jekyll – an absence of bilingualism. I want at least two languages in the blog, bit I didn’t want to hard code the “transparent” support of Russian and English. So, I added the concept of the language to each post (or a file), and the language attribute is propagated to templates. Goblog doesn’t know about particular languages, but it allows to figure it out in templates.
For example, the front page in Russian and the front page in English.
I don’t like web programming: javascript, css, html, or web design, which I cannot do at all. But I had to do it for Goblog (Octopress was much easier in this). I based my design on the Jekyll’s author site. The design is very simple. Besides, many people read the blog via RSS anyway and go the blog web site for commenting only. Hence, I need the properly working RSS feed and the post page providing comfortable reading, without fancy fonts and weird formatting.
Do you think, I’ll start convincing you to use my engine? Not at all. I tried to make it flexible and not hard coded specifically to this blog, but I had to migrate my old posts from Blogspot and comments, support two languages etc. At the result there are a couple of places still hard coded (especially in the area of Disqus links).
I can only recommend trying to implement the personal blog engine by yourself. Why? First, it usually takes a few nights only. Second, you will implement the only required bared minimum without complicated and unnecessary bell and whistles. Third, it is fun.
■
Related links:
The first edition of this book was in 1982. Old code in C, which was considered to be “good” in the past, today quite often doesn’t stand up to criticism (even though its criticising is a pointless exercise), but it’s worth to study.
I was extremely curious to have a look at puzzles in C created in eighteenth. So, fasten your seat belts. If The Hell in C exists, this is it. I’ve given a few “problems” from this book.
For me, these snippets are obvious candidates for Obfuscated C Code Context. When have to deal with such code in real life, switching to another reality is the the best solution.
After skimming the book to the end (they give the solutions there), you begin to understand its usefulness. Without seeing bad code, it’s difficult to recognize the good one.
Just have come across this book in Twitter. A simple search gives plenty of options to read it.
I wouldn’t buy it for myself because for a more or less experienced programmer it doesn’t give anything new. But if you have a hour or two, it makes sense, for a tea or a coffee, to browse and refresh in memory various quirks of pointers, memory models (thanks to the weird segmentation inherited from 8086 and 80286), ABI, peculiarities of linking in UNIX and Windows, and interesting facts from the history of the languagemand its editions over time.
This book has a little bit of everything.
P.S. By the way, I thought I know C much better than C++, just because the language is smaller and simpler. More over, I began programming in C much earlier. Thus I’d taken a Brainbench C test at some time, and it turned out that there are still many aspects of the language which I haven’t come across so far.
Agreed, the Brainbench is only a test, in many respects far from reality, but nevertheless allowing to expand the scope of self-understanding.
vector
. This function needs to return this vector to the caller. What’s the better way doing it?
From clarity perspective the best way is:
std::vector<int> create_vector(const size_t N) { std::vector<int> v; v.resize(N, 0xDEADC0DE); return v; }
Here the vector instance is being returned by value, which means potential deep copying of the object to the context of the caller.
Immediately a question raises: if the vector is huge, such deep copying can be expensive but absolutely redundant. A “more clever” approach could be:
void create_vector(const size_t N, std::vector<int>* v) { v->resize(N, 0xDEADC0DE); }
Here the vector is passed by pointer which guarantees the absence of deep copying. But this code looks quite bad.
Let’s compare the performance of these two approaches on the 100MB vector of ints. The compiler:
Apple clang version 3.1 (tags/Apple/clang-318.0.45) (based on LLVM 3.1svn)
Target: x86_64-apple-darwin11.3.0
By value:
#include <iostream> #include <vector> std::vector<int> __attribute__((noinline)) create_vector(const size_t N) { std::cout << "by value" << std::endl; std::vector<int> v; v.resize(N, 0xDEADC0DE); return v; } int main(int argc, char* argv[]) { for (size_t i = 0; i < 10; ++i) { const size_t N = 1024 * 1024 * 100; std::vector<int> v = create_vector(N); if (v[i] != 0xDEADC0DE) { std::cout << "Test is rubbish" << std::endl; return 0; } } return 0; }
Run:
clang++ -O3 -o by_value by_value.cpp && time ./by_value
Result:
0m4.933s
Now by pointer:
#include <iostream> #include <vector> void __attribute__((noinline)) create_vector(const size_t N, std::vector<int>* v) { std::cout << "by pointer" << std::endl; v->resize(N, 0xDEADC0DE); } int main(int argc, char* argv[]) { for (size_t i = 0; i < 10; ++i) { const size_t N = 1024 * 1024 * 100; std::vector<int> v; create_vector(N, &v); if (v[i] != 0xDEADC0DE) { std::cout << "Test is rubbish" << std::endl; return 0; } } return 0; }
Run:
clang++ -O3 -o by_pointer by_pointer.cpp && time ./by_pointer
Result:
0m4.852s
Time is both cases is the same. It turns out that it’s worth to choose the first one. It looks nicer and cleaner, and it’s as fast as “by pointer”.
There are two explanations. The first, probably the main one, is RVO, Return value optimization. In RVO the compiler figures out that the local instance of the vector is supposed to be a return value, and it allocates it directly in the caller context. So it doesn’t need to copy it afterwards. In fact, the compiler implements passing by reference, but it does it implicitly, without spoiling the beauty of the source code. This trick works for any classes, not only from STL.
But the optimization isn’t guaranteed thing, and there is one more tool. The standard STL containers are implemented such way that even on deep copying they only copy by value a small control structure, and the payload is “moved” by flipping pointers. It’s small overhead, but perhaps it’s worthwhile for the sake of preserving the code clarity.
Also in the context of move semantics in C++11 there won’t be unnecessary copying at all if the class is “properly” implemented (which is true for STL classes).
Conclusion: use STL containers where ever possible and trust the compiler. Sometimes, of course, the compiler is wrong, but such cases are much less than vice versa.
There are plenty of books published about writing code. Thus I was skeptical after a friend of mine showed me yet another one. To my surprise, without a long introduction it went straight to the point and coined the following statements right in the first chapter:
The style of the book was precise and concrete. I felt that in less than two hundred of pages it’s quite ambitious to cover the subject of “the art” and decided to order the book to find out how they’re going to do that.
Eventually, a few hours of reading on weekend were worthy. Though an experienced programmer will barely find any eye openings in the book, but surprisingly, this is a compact, concise and solid handout for juniors. Without too much theory, always using real examples, the authors go through many key points of writing code: how to name variables, functions and classes, how to structure the code, how to deal with efficiency-readability trade off, how to comment, where to compromise and where be a perfectionist. Again, it’s all in less then two hundred pages. Plus they briefly touched unit testing.
The authors not only tell you what is good and bad, they always demonstrate “why” on examples gradually improving “regular” code. At the end they put a real example of a class counting network traffic and returning a number of bytes transferred in the last hour and day.
They began with a naive implementation and then worked through two more versions showing that sensitive balance between efficiency and readability. I think that even experienced developers may find this example interesting to play with.
To sum up, this book can fit perfectly into your team book shelf and be used as a quick reference of how-tos. Buying for yourself is questionable, because at home you’d probably prefer something more fundamental.
It could be fixed this way:
void f(int a, int /* b */) { ... }
But it looks ugly.
There is a better way:
#define DISCARD_UNUNSED_PARAMETER(x) (void)x void f(int a, int b) { DISCARD_UNUNSED_PARAMETER(b); ... }
This macro is clear, and you can easily find all such places in the project.
By they way, in Go
this situation is treated as the error, not warning. It can be annoying when you add, remove, comment out and put back things all the time, because sometimes you do have unused stuff. But eventually such approach doesn’t allow that temporary garbage to spread and stay in the code (for example, in C++ who on Earth wants to clean up the list of included STL headers after hours of coding?)
<script language="javascript"> (function() { function async_load(){ var s = document.createElement('script'); s.type = 'text/javascript'; s.async = true; s.src = 'URL of a script to load'; var x = document.getElementsByTagName('script')[0]; x.parentNode.insertBefore(s, x); } if (window.attachEvent) window.attachEvent('onload', async_load); else window.addEventListener('load', async_load, false); })(); </script>
They say, this is a modern HTML5 compatible method, the idiom.
Why?
I’ve decided to change the approach and implemented Google’s way: Don’t sort. Search!
Now, on the main page there is a search. The search allows to filter out the posts containing particular word(s) in the title or body.
The search is based on the reverted index, without sorting by relevance. The posts, containing required words, are displayed sorted by default by date.
Try to search for “maximite” or “ruby nor”. If there are several words are given in the search, it’ll show the posts containing all these words. The words, which are shorter that 3 characters, are ignored.
For me - this is very convenient!
Checked in Chrome and Safari on Mac.
The idea was previously described in a series of posts:
The article discovers an implementation in Ruby. Now NOR assembly code looks almost as a regular assembler language. I was amazed how powerful Ruby is in creating domain specific languages (DSL). Mostly because it allows to omit parentheses in function calls.
void Test_SplitPair() { typedef std::pair<std::string, std::string> Pair; using string::SplitPair; const Pair p1 = SplitPair("", '='); assert(p1.first.empty()); assert(p1.second.empty()); const Pair p2 = SplitPair("=", '='); assert(p2.first.empty()); assert(p2.second.empty()); const Pair p3 = SplitPair("name=value", '='); assert(p3.first == "name"); assert(p3.second == "value"); const Pair p4 = SplitPair("name = value", '='); assert(p3.first == "name"); assert(p3.second == "value"); const Pair p5 = SplitPair(" n ame \t = va lue \r\n", '='); assert(p5.first == " n ame \t "); assert(p5.second == " va lue \r\n"); }
Any ways to make to better? The ideal solution is to split to multiple tests. But even less radical approach works to avoid copy-paste problems:
void Test_SplitPair() { typedef std::pair<std::string, std::string> Pair; using string::SplitPair; { const Pair p = SplitPair("", '='); assert(p.first.empty()); assert(p.second.empty()); } { const Pair p = SplitPair("=", '='); assert(p.first.empty()); assert(p.second.empty()); } { const Pair p = SplitPair("name=value", '='); assert(p.first == "name"); assert(p.second == "value"); } { const Pair p = SplitPair("name = value", '='); assert(p.first == "name"); assert(p.second == "value"); } { const Pair p = SplitPair(" n ame \t = va lue \r\n", '='); assert(p.first == " n ame \t "); assert(p.second == " va lue \r\n"); } }
In my case it was “Seven languages in seven evenings”. For each language there is an introduction, which makes sense only if a language is brand new for you. There are also interviews with creators of the languages. One of the asked interesting questions was about what the author would like to change in the language if he could re-design it from scratch now.
Languages:
The reviews of each chapter below are my subjective views two things at once: a programming language and a material about it. Why? For familiar languages it hardly makes any sense to describe the language per se, but to note interesting distinctive features could be useful. But if a languages is a green field, it is worth describe it in general.
Ruby
The Ruby chapter was quite useless for me because I thoughtfully read “Programming Ruby 1.9”, and have been hooked. Ruby is an amazing scripting language. Each time when programming in Ruby I feel so delighted similar to when I tried PHP first time after Perl.
Ruby’s creator, Yukihiro Matsumoto, says in the interview, that if he could re-design Ruby today, he’d like to change the concept of multi-threading to Actor.
In short, “Actor” is when concurrent threads don’t share memory and don’t use mutex or semaphores for synchronization. Instead, they send and receive messages to each other, and messaging is provided by runtime and built-in to the languages syntax. Examples: Scala, Go, Erlang, Io.
Io
Io is a very small, compact language, based on prototypes like JavaScript, where there is no distinction between classes and objects.
There is an interesting concurrency feature in addition to actors and coroutines (cooperative multi-threading as in Lua), called futures. “Futures” are similar to the actor. There the only difference is when the caller thread tries to use the result of the future, it will be blocked until the future completes and gives the result back.
An example from the book:
// Fire up the future. futureResult := URL with("http://google.com/") @fetch writeln("Continue immediately when future is running in background.") // This line will be executed immediately after spawning the future. writeln("fetched ", futureResult size, " bytes") // But this line will be blocked until the future returns.
Prolog
I’ve been gnawing this animal for years. But thanks to Erlang recently, all this functional stuff in general is now giving up for me, and monsters like Prolog or Haskell don’t look so scary anymore.
It turned out that the depth of the material about Prolog has matched precisely with my level. The eight queens problem and a Sudoku solver were excellent examples for me.
Shortly, a program in Prolog is a pile of facts and rules. Then the Prolog runtime performs a depth-first search amongst possible results and tries to find those satisfying all the given facts and rules.
In fact, the Sudoku solver program is a set of variables, representing the Sudoku field, and a list of rules (summations by columns, rows and squared groups) according to the rules of Sudoku. Then Prolog performs an exhaustive search to find the values and their combinations satisfying the rules.
Of course, this is very much a superficial glance, but this has given me much more understanding of Prolog.
Scala
I will note only a few facts interesting to me.
Multi-threading is based on actors. After Erlang and Go you understand how good and handy it is.
I think that Scala has all possible bells and whistles even invented for programming languages. But sometimes it has not only good consequences.
Erlang
I’m a big fan of Erlang and already read a few big books. That’s why this book hasn’t given to me anything new. But for novices this introduction may give quite solid view on Erlang functional approach and concurrency model.
Clojure
Clojure is a Lisp-based language driven by Java VM.
It has an interesting feature called STM, software transactional memory. In STM a piece of code is declared to be a transaction. It is executed atomically or all the changes of variables are rolled back.
And finally, Haskell
Haskell is a taught guy. The introduction in this book is very light and minimal, just to remember the word “Haskell”. I read “Programming in Haskell” and currently I’m on “Real World Haskell”, that’s why I simply skimmed the chapter in this book.
Okay, to sum up. This book is to be read just once, to broaden your outlook.
Simply wanted to share current interests.
Everyday work (design, planning, coding and reviews): C and C++. C++ 0x11 goes at full speed and it is worth to catch up. ACCU 2012 is almost fully dedicated to the new C++.
For fun:
To “gnaw” in hope to write something real - Haskell and Prolog.
In a queue at least minimum acquaintance: Clojure. This one is more promising because it is Lisp.
After overall migration to Mac, I wish to try Objection-C and AppleScript in action. But what to write on Mac, in Objective-C? Of course, UI! But UI is totally out of my interests. But, Objective-C still looks tempting because of so quickly growing mobile apps market.
From recently touched, but not involved:
So, the connections are determined and now we have to formalize and visualize them.
At the first glance this is not an easy task, but it turned out there is a simple and elegant solution.
There is a plain text language to declare graphs – DOT. Its beauty is in ultimate simplicity. For example, a trivial graph:
graph name {
a -- b
b -- c
b -- d
}
Feed it to special software and get this:
That’s it! The output is in SVG, ready to stick on a wall.
Unfortunately, the best software I’ve found to visualize DOT is Graphviz. It does pretty decent job properly processing quite sophisticated graphs, but in terms of user experience it’s shite.
If anyone is interested, I’ve shared a real trace (obviously, names are obfuscated). It gives an idea about simplicity of the source and visualization capabilities - PNG and SVG.
Again, the graph formalization is dead simple - you only need to specify pairs of connected vertices. Also, in DOT you can describe directed graphs and extra attributes of the vertices.
To sum up, great technology.
This is a micro-computer based on Microchip PIC32 running BASIC. It is so simple that even a novice can build it in a few hours.
It is a bit more powerful than Radio-86RK and ZX Spectrum 48. But its peripherals are fantastic: SD/FAT card, USB, VGA, PS/2, timers, RS232, I2C, SPI, PWM, ADC/DAC and individual general purpose pins.
If you build it on a mock up board buying parts by yourself, it will cost less then ten Australian dollars.
The project is open-sourced (schematics, PCB artwork, sources).
Even if quickly flip through the documentation, no doubts - a list features is impressive. All peripherals are available directly from BASIC.
Programs and data can be stored on a SD card. If there is a “AUTORUN.BAS” file on the card, BASIC will run at the start.
I liked Maximite, but soldering is not my favourite activity. Unfortunately it is only possible to buy a kit, but not a fully assembled unit.
I ordered the kit from Altronics and soon after it arrived.
Only the microprocessor was already soldered because soldering such form factor chip isn’t an easy task.
Anyway, screw it, let’s do it.
Here a few parts are already in place. I’m not a complete newbie in soldering but last time I took a soldering iron in my hands was about five years ago. I had no acid for soldering, so I was crumbling rosin right to soldering points. The effect is similar to acid. The soldering iron (the one on the plate) with a sharp sting.
I spent the first hour struggling with only a few parts, but eventually it went smoother.
A half is ready.
After one more hour it was all done.
Maximite can be powered from an external 9V source or form USB. I used USB.
Plug into USB and VGA. An off you go!
BASIC is ready but there is no keyboard. I hand’t a proper PS/2 one and I tried a USB-PS/2 connector. Alas, it didn’t fit.
Next day I found the PS/2 keyboard and finally connected.
The case.
Fully assembled.
I have to admit - the kit from Altronics is a very good quality product. The holes are metallized on the board, and the case fits perfectly.
Then I had to upgrade the firmware to the latest version. Maximite can flash itself over USB without a special programmer. Just open the case and hold a special button when switching Maximite on. It goes to a boot loader mode.
Maximite is a standard CDC device in USB infrastructure. Windows still requires a driver though, but Mac has it built-in.
Plug-in.
Flashing.
Right, the firmware is upgraded up to 3.0A.
As I said, Maximite supports VGA and PS/2, but you can also connect it to a PC via USB. In this case Maximite mirrors the VGA output to that serial connection to the PC, also treats the data coming from that connection as keyboard input.
So, it is possible to detach VGA and PS/2 at all and talk to Maximite over the serial USB connection only.
For example, VGA dispay (voltmeter):
The same data on in a terminal emulation application on the PC:
Interestingly, Maximite works with pixels, not characters. So when a character is being displayed, it is also copied to the console, but when Maxitile draws graphics it is not visible in the serial console.
BASIC language in Maximite gives full control over the peripherals using operators.
There is an archive of BASIC programs running on Maximite available of the project website.
There are a few screenshots.
Clock.
Character editor.
Voltmeter.
Puzzles.
I believe, no comment here.
Conclusion
Every penny I have spent on Maximite is worth that fun I’ve got.
The Maximite project is surprisingly solid. Everything is nice and simple. And it works!
For beginners, even kids, interested in microelectronics, Maximite is simply a godsend. Easy and nice to build. I as an amateur have built everything in a few hours only.
When my brother was building Radio-86RK Emulator and Spectrum about twenty years ago, there was a joke about DIY projects published in radio electronics magazines: if a author says that his device doesn’t required any tuning, there is at least a little chance to get it working; but the author says that his device does require some minor tuning…
Anyway, if you want to come back to your 8-bit youth with a soldering iron in hands — build Maximite.
An idea — periodically take a snapshot of the process stack trace. For instance:
#0 0x991a8c22 in mach_msg_trap ()
#1 0x991a81f6 in mach_msg ()
#2 0x968870ea in __CFRunLoopServiceMachPort ()
#3 0x96890214 in __CFRunLoopRun ()
#4 0x9688f8ec in CFRunLoopRunSpecific ()
#5 0x9688f798 in CFRunLoopRunInMode ()
#6 0x92158a7f in RunCurrentEventLoopInMode ()
#7 0x9215fd9b in ReceiveNextEventCommon ()
#8 0x9215fc0a in BlockUntilNextEventMatchingListInMode ()
#9 0x90010040 in _DPSNextEvent ()
#10 0x9000f8ab in -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] ()
#11 0x9000bc22 in -[NSApplication run] ()
#12 0x902a018a in NSApplicationMain ()
#13 0x0012e356 in main ()
For Linux, HPUX and Solaris there is a tool called pstack
, and procstack
on AIX. I’m sure it is possible to do the same on Windows because Process Explorer can do it.
Comparing the current stack trace with the previous one we can measure how much it has been changed. If the stack wasn’t changed at all or only a few deepest lines were changed (for example, inside the kernel), we may assume that this process is stuck. The deadlock on a file or database is even simpler because the code will be blocked on a function inside the kernel.
Of course, such detector has to be adjusted for specifics of the monitored processes. But it can be configurable via regular expressions or a script language as Lua, for instance.
The good thing is that such monitor doesn’t require any changes in the target software, and can be implemented on any language suitable for easy text parsing, for example, Ruby or Python.
Am I reinventing a wheel?
void foo(T* t) { bar(t); }
The problem is that the function bar
is a part of a legacy library which we cannot refactor right now. The signature of bar
is void bar(T*)
. T
is not const
. But in reality bar
never changes an object referenced by t
. This is how it was implemented.
But foo
is a part of a brand new API, and we want to make nice or clean. The contract of the function foo
says that it doesn’t need to change its parameter.
I think the code should be like this:
void foo(const T* t) { bar(const_cast<T*>(t)); }
Why? The contact of foo doesn’t require the pointer t to be non-const. We must reflect this in the API by making t
const
. It doesn’t matter that for some reason a particular implementation of foo
is based on the legacy bar
function not having const
in the argument but never changing it. Yes, we have to use the ugly const_cast
but this bad code is nicely isolated inside foo
only and doesn’t affect our nice and clean brand new API. Moreover, if we refactor foo
at some point and get rid of legacy bar
at all, the problem will disappear completely.
Here is a counterargument from my colleague: it may turn out that the function foo
can have a bug and accidentally change t
even it is declared as const
. The solution is to simply keep the argument of foo
non-const. In this case we don’t need that cast, we explicitly show to an end user of foo
that she should expect its parameter to be const
, and eventually we never violate the contact of the function foo
.
Eventually we haven’t agreed. My flaw is that const
doesn’t really protect from side effects coming from legacy bar
and the argument of foo
may be changed regardless being const
. My friend’s flaw is that it is not easy to explain in the documentation how and why the argument of foo
may be changed. Just because our particular implementation dictates this? Such approach spreads the drawback of the legacy code to our nice and shiny new code.
Dilemma.
P.S. There is another esoteric approach — to create a temporary deep copy of T
inside foo
and pass it to bar
by non-const pointer. Personally if I have to choose between quick but badly designed code and slow but nicely written code I usually go for the second one. Tomorrow we can buy another faster computer and the slow code will be faster, but that computer will make the bad code better.
I thought that biographies are for “old people”. This book has changed that mindset in me. Being driven by massive ads and recently emerged love with Apple products and philosophy I had started reading.
Initially it was like an expanded version of “Pirates of Silicon Valley” which I, frankly, liked, but as reading progressed I had just fallen in love with this book and weren’t able to stop.
The book can be treated as conjuncture or just an attempt to leverage the situation that Steve has just passed away. I was fully biased but after the first chapter I was hooked.
It is fascinating reading. Maybe just because the author is talented, I don’t know.
Conclusion: strongly recommend.
A recent.
“The world is flat 3.0: A Brief History of the Twenty-first Century”, Thomas L. Friedman
This is a killer book to understand or even discover that nowadays:
And many other topics.
Frankly, there were moments in my life when I doubted and thought I would be better off doing, for instance, real estate, natural resources, ads or media rather than software. If you have even a tiny similar concern this book will wipe it out forever, and maybe even show the way.
To conclude: very useful, interesting and fascinating reading.
I wrote this code:
#include <iostream> class A { public: void bar() { foo(); } private: virtual void foo() = 0; }; class B: public A { private: virtual void foo() { std::cout << "B::foo()" << std::endl; } }; int main(int argc, char* argv[]) { A* a = new B(); a->bar(); delete a; return 0; }
VS2010 and GCC compile it perfectly and it prints out B::foo()
.
I have concluded that the virtual function mechanism usually implemented via vtable is runtime, but public/private is compile time, and they don’t depend on each other.
Artemy Lebedev said that he despises all this SEO bullshit because it is much more productive to focus on creating great posts rather than thinking how to attract more readers. His LJ uses a default template without even minor tweaks to make it looking “cool”. Despite of this his blog is in Russian Top 10.
Nobody reads blogs directly nowadays. Everybody uses Google Reader or other aggregators making the design of blogs absolutely pointless.
Anyway let’s leave odious Lebedev and come back to our techy blogs. Tom Preston-Werner, one of the Github founders, said that he wanted to write posts but not tweaking different templates or keeping the version of WordPress or Mephisto up to date.
A similar thought haunts me from the day one of using Blogspot. Only recently I have sorted a way of creating post by using ReST. Now I write posts in this markup and convert to HTML before publishing by a few handcrafted scripts. Unfortunately hosting for images is not automated in anyway.
Many times I wanted to migrate to a standalone platform, for example, WordPress. But my laziness and unwillingness to spend even a second on its maintenance always stopped me.
At the moment I have another attack of hate to Blogspot and as a consequence look for alternatives.
Surprisingly I never considered using static blog engines.
What if just simply migrate to GitHub Pages? It converts one of your repositories to a website which can be also processed by Jekyll, a static engine from Tom Preston-Werner.
I could kill a few birds with one stone – to use a proper markup language, Markdown or Textile, instead of bloody HTML and to manage publishing by git. Of course ReST is better but I can cope with it.
Also GitHub Pages can be integrated with your own domain if needed.
Comments and discussions. Disqus seems to be an easiest way to sort it without any hassle.
Google Analytics perfectly works with any engine where you can insert their JavaScript hook into pages.
What is still missing?
Sometimes people still visit blogs directly, for instance, to find previous posts or explore “social” details. In this case all these bells and whistles in a form of JavaScript gadgets are very useful and save a lot of time. Blogspot allows adding them in a few clicks.
But coming back to the question at the beginning – do I really need them? If the content is interesting the audience will inevitably find you. A word of mouth in a form of Twitter or Facebook will attract people. But if the content is crap all those bells and whistles (and SEO) will not help.
I went to check out the stop and the Chromebook.
The “Shop” is a two desk place with a dozen of demo notebooks in the PC World nearby Apple, Sony and other departments.
The Chromebook demoing there is Samsung Series 5 3G.
Below there are a few not quite good quality pictures I’ve done during an hour playing with this device at the shop.
Closed lid.
Left - power, a weird video connector, USB and audio.
Right - one more USB.
Front - SD.
The power socket is quite fragile.
Look at the keyboard.
Above Shift there is an interesting button called “Search” opening a new tab and pointing the cursor to the URL bar. In fact, CTRL-T. I liked this button.
There are internet related buttons in the top row. The button between full screen and the brightness is the switch amongst Chrome windows when there are multiple of them opened.
There are no Ins, Del, Page Up and Page Down.
After 8 seconds of boot time you see a login window for your Google Account (I used a demo one), and then the main screen comes out.
In Chromebook there is only one application - the Chrome browser. It is not possible to exit from it. You can only open new tabs and windows.
The Chrome itself differs from its desktop version. There are extra menus. For example, network settings.
WiFi
Russian keyboard layout is configurable through the settings.
Printing works via a direct connection to a USB printer (HCL is provided), or a “proper” Google way printing via Cloud Print.
It is possible to keep files locally. For instance, you can attach them to e-mails.
Only a few types of files allow to do anything with them. For example, photos and vides can be uploaded to WebPicasa.
All downloads and screenshots also go to the local storage.
Skype was not available, so I tried GTalk (this is mine shaved head in the window).
GTalk settings.
After a few minutes of jiggling around GTalk crashed.
Of course, I checked out the right website.
And another one.
And another one to figure out the overall performance.
Linux booted in 33 seconds. On my Mac Air Core Due it boots in 8 seconds. But bogomips were the same (~20) for some reason.
Now my biased conclusion.
400 quid is nuts. The notebook itself isn’t light, tiny and slick, which could be an excuse for such high price, but there are a dozen of other netbooks around doing the same stuff, plus the rest.
Even at the shop, where WiFi was really fast, working fully online becomes annoying very soon due to network lags. The official “Offline Gmail” is available but it is a joke, and barely usable seriously.
For 50 quid I could buy it right there, just because I use Google’s products and is willing to play with this “Google Console”, but 400 is “no, thank you”.
void* p = (struct this_does_not_exist *) -1;
Remove struct
, compile as C++ and get an error:
cast.cpp
cast.cpp(1) : error C2065: 'this_does_not_exist' : undeclared identifier
cast.cpp(1) : error C2059: syntax error : ')'
Adding a forward declaration:
class this_does_not_exist; void* p = (this_does_not_exist *) -1;
And it again compiles cleanly.
All examples conform the Standard but to be honest the first one is really odd.
GCC gives a warning at least.
Every time I come across famous Shakespeare’s trade off “to refactor or not refactor” I come back to this picture. You can keep going to put more and more cheap crutches preventing your house from irreversible falling down. It can help for sometime but at some point there will be a point of no return. Or you can get all the people out of the house for a while and rebuilt it of bricks.
#include <string.h> #include <stdio.h> int main(int argc, char* argv[]) { char b[32]; strcpy(b, "123456789012345"); strcpy(b + 1, b); printf("[%s]\n", b); return 0; }
There is a problem here because the parameters of strcpy()
overlap.
This is an unpredictable behaviour because strcpy()
doesn’t guarantee the order of moving bytes (from left to right or vice versa) but the result depends on it.
Check on different compilers and platforms.
Visual Studio 2010 64-bit
[1123446788012245]
The result is corrupted every four bytes. Obviously, it has been copied 32 bit words.
Linux 64-bit
[1123456788012345]
The result is now different. Compiler and libc:
ldd --version
ldd (GNU libc) 2.5
gcc --version
gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-50)
man strcpy
says:
The strings may not overlap...
Why not “must not”?
Solaris (SPARC)
[1123446788012245]
Compiler and libc:
cc -V
cc: Sun C 5.8 2005/10/13
version /usr/lib/libC*
version of "/usr/lib/libC.so.3": SC2.0.1 12/20/94 Sun C++ 3.0.1 patch 100962-09
version of "/usr/lib/libC.so.5": Sun SUNWlibC SunOS 5.10 Patch 119963-06 2006/04/21
version of "/usr/lib/libCrun.so.1": Sun SUNWlibC SunOS 5.10 Patch 119963-06 2006/04/21
version of "/usr/lib/libCstd.so.1": Sun SUNWlibC SunOS 5.10 Patch 119963-06 2006/04/21
AIX
[1111111111012245]
This result is clearly wrong. The man pages are pretty clear on it:
String movement is performed on a character-by-character basis and starts at the left. Overlapping moves toward the left work as expected, but overlapping moves to the right may give unexpected results.
Compiler and libc:
lslpp -L | grep Compiler
vacpp.cmp.core 8.0.0.20 C F IBM XL C/C++ Compiler
lslpp -L | grep libc
bos.rte.libc 5.3.9.1 C F libc Library
HP-UX
[1123456789012345]
Compiler:
what `which cc`
HP C/aC++ for Integrity Servers B3910B A.06.22 [Nov 14 2008]
This result is correct but man pages warn in a funny way:
Character movement is performed differently in different implementations, so moves involving overlapping source and destination strings may yield surprises.
Conclusion: strcpy()
is bad, due to many reasons.
In this example n=5, m=4, ten edges: 1-1, 1-2, 2-1, 2-2, 3-3, 4-1, 4-3, 5-1, 5-2, 5-4, a number of crossings: 10.
Crossings are always considered in pairs. For example, if three edges are crossed in one physical point, formally there are still three crossings, not one.
O(n*m) solution exists.
For example, it can be done this way:
#define arraysize(array) (sizeof(array) / sizeof(array[0]))
But there is a little problem over there. If accidently a pointer is passed to the this macro instead of an array, the code still compiles but the value will be far from being conceived.
There is a way to make this macro safer.
template <typename T, size_t N> char (&ArraySizeHelper(T (&array)[N]))[N]; #define arraysize(array) (sizeof(ArraySizeHelper(array)))
Looks cryptic, but we can break it apart:
T (&array)[N])
- an array definition (T array[N]
) passed by referencechar (&ArraySizeHelper(...)[N]
- a function returning a array by referencesizeof(ArraySizeHelper(array))
- take a size of the function return value typeFrankly, it is not easy to get it. But this macro is great.
By the way, we can play with sizeof()
of the function return value type:
#include <iostream> #include <string> std::string f() { return std::string(); } int main() { std::cout << sizeof( (&f)() ) << std::endl; std::cout << sizeof( std::string ) << std::endl; return 0; }
My VS2010 prints out “28” twice.
Interestingly, in C it is also possible:
#include <stdio.h> struct t { char x[1024]; }; struct t f() { struct t a; return a; } int main() { printf("%d\n", sizeof(struct t)); printf("%d\n", sizeof( (*f)() )); return 0; }
It prints out 1024
twice.
#include <stdio.h> int main() { int i; for (i = 0; i < 8; ++i) printf("%c", "12345678"[i]); printf("\n"); return 0; }
To me personally the expression "12345678"[i]
cuts the eye. But from the language point of view everything is fine.
Scott Meyers
Move semantics, perfect forwarding, and rvalue references
John Lakos (Bloomberg)
Defensive programming done right
Dietmar Kuhl (Bloomberg)
Generic Programming with C++ 0x
Other videos are also available.
But today, thumbing through “Windows Internals”, I came across a paragraph about it.
So, there is a little log from the console (Windows 7).
ver
Microsoft Windows [Version 6.1.7601]
Create a file and a directory:
cd C:\Temp\links C:\temp\links>mkdir folder C:\temp\links>echo >file
Create a symbolic link to the directory:
C:\temp\links>mklink /D link1 folder
symbolic link created for link1 <<===>> folder
Create a junction to the directory (it isn’t possible to point it to a file):
C:\temp\links>mklink /J link2 folder
Junction created for link2 <<===>> folder
Create a symbolic link slightly differently:
C:\temp\links>mklink link3 folder
symbolic link created for link3 <<===>> folder
Create a symbolic link to a file:
C:\temp\links>mklink link4 file
symbolic link created for link4 <<===>> file
Result:
C:\temp\links>dir
Volume in drive C has no label.
Volume Serial Number is C021-6C9F
Directory of C:\temp\links
09/05/2011 18:26 <DIR> .
09/05/2011 18:26 <DIR> ..
09/05/2011 18:26 13 file
09/05/2011 18:25 <SYMLINKD> link1 [folder]
09/05/2011 18:25 <JUNCTION> link2 [C:\temp\links\folder]
09/05/2011 18:25 <SYMLINK> link3 [folder]
09/05/2011 18:26 <SYMLINK> link4 [file]
09/05/2011 18:23 <DIR> folder
3 File(s) 13 bytes
5 Dir(s) 208,278,925,312 bytes free
Note the interesting types of files: <SYMLINKD>
, <JUNCTION>
, <SYMLINK>
. The book says the first two are identical in functionality, simply <JUNCTION>
is older mechanism available in older versions of Windows and supporting links within the same volume only.
Also, note that even link3
points to a directory, it doesn’t behave as a regular directory (in contrast to link1
and link2
which work normally as directories). FAR, by the way, also doesn’t see link3
as a directory.
In general, such simple task as links in the file system, solved in UNIX more than twenty years ago, has been solved in Windows in traditional for this operating system way - there are multiple solutions with different level of compatibility.
By the way, “Windows Internals” is bloody fantastic, strongly recommend.
We fixed that bug (including Makefile problem) but a generic question occured: what exit code is returned to the parent when the process crashes before executing exit()?
In UNIX there are special macros inspecting the status returned from wait(). But all UNIXes are different and there is Windows as well.
Eventually I wrote a simple self-killing program:
#include <stdio.h> #include <stdlib.h> #include <string.h> int main(int argc, char* argv[]) { char cmd[80]; int r; sprintf(cmd, "%s ?", argv[0]); if (argc > 1) { if (argv[1][strlen(argv[1]) - 1] == '1') *(char *)0 = 0; exit(0x77); } printf("Normal: %08X\n", system(cmd)); cmd[strlen(cmd) - 1] = '1'; printf("Crash : %08X\n", system(cmd)); return 0; }
And run it on different systems.
Windows 7, Visual Studio 2010, cl crash.c && crash
:
Normal: 00000077
Crash : C0000005
Linux x86_64 (cc -o crash crash.c && ./crash
):
Normal: 00007700
Crash : 0000000B
Signal 0x0B (13), by the way, is SIGSEGV
, segmentation violation.
Solaris SPARC 5.10:
Normal: 00007700
Segmentation Fault - core dumped
Crash : 00008B00
HP-UX Itanium 2:
Normal: 00007700
sh: 25112 Memory fault(coredump)
Crash : 00008B00
AIX 5.2:
Normal: 00007700 Crash : FFFFFFFF
Here seems the exit code wasn’t even propagated to system()
.
Conclusion: it (as usually) really depends on the operating system.
At the moment I’m working on a project customizing putty to support of-the-fly syntax highlighting for a proprietary programming language. The main difficulty is that the language has very complicated and not well defined and irregular grammar with lots of ambiguities.
As a result what I’m doing is fixing different special cases here and there. But syntax highlighting is complicated in general - you fix one thing and easily break ten others at the same time.
After some time of maintenance I gave up and I spent a day setting up cmockery and all the rest of unit testing plumbing. Then I remade all code examples I used to verify everything manually in a form of unit tests.
Once I’ve done that life become simpler. Now after every change I can automatically check by running tests that all previous cases are not broken. I’m confident to change things without being afraid of regression. Of course I spent an extra day to two developing, debugging and configuring unit tests infrastructure but it has paid off already, fully or even more.
Every new feature or bug fix I start from creating a test demonstrating how it should work, and, obviously, this test must fail. Then I implement a bit of functionality to make this test working. I cannot image how I might develop this project further without unit tests.
By the way, this is a classical approach when tests are written beforehand. And the main code complements the tests afterwards.
Thus, there are ten common misconceptions.
1. The amount of money stored on the card.
On regular credit or debit cards (even having a chip) there is no money counter. The card is just an identifier. There are exceptions in the form of additional wallet applications on the chip cards. Usually it can be discount or loyalty programs, virtual money (for instance, petrol liters) etc. In general this is not related to the normal card usage. Such applications are usually accepted in special stores that only support these cards.
2. Anyone, who wants to accept the plastic cards, can be connected directly to Visa, MasterCard or to other international payment networks.
It’s not possible for just anyone to connect directly to Visa or MasterCard. Only major banks or independent processing centers can do it because it requires special equipment, considerable insurance accounts, security certification and lots of other “little things”. Furthermore not every bank can afford it. All others wishing to take cards use their services.
3\ ATMs or POS terminals are connected directly to Visa or MasterCard.
Major international payment networks do not have their own ATMs or payment terminals. ATMs or POS terminals must belong to a bank, which, in turn, is either directly or indirectly (see #2) connected to the payment network.
4. I have $200 “on the card”. That’s all that I can spend.
The card balance and the amount you can spend daily using the card are not entirely related to each other. It is more constructive discussing the daily limit on the card. The daily limit depends on many factors, and can be either less than the card balance or even greater. For example, even having one million on the account, you may not be allowed to withdraw more than a few thousand in cash a day at the ATM (and this is not an ATM hardware limitation). And vice versa, if you are a VIP customer, usually having millions in the bank, but at the moment you’ve lost everything in a casino, after the call to the bank, individually, some of the top managers can give the command to increase your daily limit allowing you to pay off. In this situation the bank takes all the responsibility that you will return everything back.
5. ATM or POS terminals validate the PIN.
In the overwhelming number of cases, any use of the card refers to a connection with the bank that issued the card. If you insert a card issued by HSBC UK in Australia, an approval to withdraw money will be requested from the HSBC UK anyway right before your eyes. This is because the PIN can be verified only by the bank that issued the card. The only exceptions are cards with a chip. Such cards can verify the PIN without connecting to the bank. These cards are microcomputers able to calculate crypto functions. Sometimes when making a purchase (not cash withdraw), a store may not connect to the bank when a non-chip card is being used if the amount is less than a certain limit. The reason for this is the transactional and communication costs. When the amount is not significant the risk of loss after possible fraud is also not significant.
6. The PIN is stored on the magnetic stripe and any bank or store employee can “steal” it when holding your card while you turn away.
In fact, there is only a crypto convolution of the PIN recorded on the magnetic stripe, and it is calculated using a crypto key stored in a bank HSM (high security module). Using data from the magnetic stripe can only verify the PIN, and only if you know the secret key. Usually the 3DES algorithm is used. HSM is a hardware device for storing keys and cryptographic operations using those keys. After initial input of keys to HSM (personalization) they are never sent outside the physical case of HSM in the plain form.
In addition to the major effort for the physical protection of these devices, they themselves are protected from an unauthorized intrusion. For example, if you try to open the case connecting a “sniffer”, all the keys will be automatically erased.
There is an interesting technique of primary keys input. For example, here is a realistic scenario. We choose N security officers of the bank, for example 3 (ideally, they shouldn’t know each other personally). Each officer generates a key and doesn’t show it to anyone else. Then, they in turn go into the room with HSM and enter their keys. After all the keys are entered, the HSM computes bitwise XOR amongst them and keeps the result as the master key. It turns out that nobody knows that master key at all. To restore the master key you have to know all original components from those N security officers, who must take care of keeping it secret.
There are no “unimportant” questions in the security, and such procedures are mandatory when the power of cryptography ends and the human factor starts.
This is very important: nobody from the bank staff, ever, under any circumstances will ask you the PIN. However, you cannot imagine how often countless customers, when being asked by the bank operator the security question say the PIN.
7. When making the purchase, money immediately comes directly from the customer’s account to the account of the merchant.
Usually clearing happens in the end of the business day. When making the purchase the amount from the available day limit (see #4) is only taken on hold. The account is debited a few days later when the bank issued the card receives the financial representation from the bank that accepted the card and processed the payment (bank-acquirer).
8. The amount, printed on your receipt when paying by card, is precisely what will be deducted from your account.
In fact, the authorized amount may differ significantly from the amount that was withdrawn by the financial transaction. It often happens in car rentals and hotels. Such merchants are allowed to charge extras (for example, for petrol and minibar). And these are not the only type of merchants allowed to adjust the final amount.
Also the amount taken on hold on purchase and the amount eventually withdrawn from the account can differ if the account currency is different to the transaction currency. Clearing usually takes a few days but the exchange rate could change in this time.
9. The amount, taken on hold when paying by card, will be withdrawn from the account anyway.
The amount on hold may be never debited from the account. After 10 (for ATM) or 45 (for other types of terminals) days, if there is no financial representation received from the bank-acquirer, the hold will be lifted. This is “good” and “bad”. It is “good” if you have just paid but want to cancel. You can contact your bank immediately, explain the reason for the cancellation, and if everything is okay with the payment, the operator cancels the transaction and the hold disappears. In this case if your bank receives a financial representation of this payment from the merchant (in a few days period) the bank will be dealing with this problem without you (and your money). It is “bad” if you waited a couple of days and the financial representation has been delivered before your call. In this case it will be much harder to cancel the transaction and get your money back. The bank will have to start an investigation, which can take up to 45 days. Meanwhile that amount will still be on hold.
10. Debit card users cannot go to overdraft.
As explained in #4, the purchase authorization logic is not based on the account balance, but on the daily limits. And it works similarly for debit and credit cards. The bank can adjust the daily limit, slightly exceeding the account balance, even for debit cards.
Hope all this information will help you avoid some unpleasant surprises when using plastic cards.
Not a problem when a client workstation runs Windows. You just go to a web site and logon, and then an ActiveX component gets started from the page, installs and runs everything required.
It becomes complicated if the workstation is not Windows. Indeed, there are RDP clients available for Linux and Mac, but it doesn’t help much because without that encrypted channel it just doesn’t have any meaning.
What do Linux and Mac users have to do? Install Windows in a virtual machine and run RDP in there. This is a good solution but with a drawback. A virtual machine hosting standard Windows usually takes a few gigabytes minimum and boots considerably slow.
I googled a bit and found MicroXP. This is a very stripped-down version of XP SP3. The distribution ISO takes around ~100 MB, and when installed ~250. In VirtualBox on Mac Air the installation lasts five minutes and the already installed system boots in 10 seconds or less. The virtual machine requirements are minimal – a dynamic 300 MB disk and 64 MB of RAM.
After the installation of MicroXP you have to add Virtual Box Guest Additions (~200 KB) to share directories and not to struggle with the mouse, and then add a RDP client (it can be transferred to the VM via the shared folders).
In summary, if you need minimalistic and fast XP, MicroXP is a very good candidate.
P.S. Don’t visit microxp.org. This is a fake to fish e-mails.
Note: Quoted materials below are provided “as is” in the original author edition. Only minor formatting changes are applied. Of course, the texts on Russian are translated to English.
But the reality was different. Literally a few hours after the announcement, I received an e-mail from Vasiliy Artemev.
Secret code: 139471
That was the correct answer. Vasiliy was a winner and received the 100$ prize. He kindly shared that he bruteforced the problem. In fact, a simple bruteforce attach forced the program printing out the magic message even on a 3-character attempt. Alas, my hash function used to check the password had many collisions and become a weak link.
But there was no complete analysis of the password checking algorithm, and Vasiliy offered to sponsor the rest of the challenge to figure out the algorithm with 50$ from his prize. The challenge kept going.
In meanwhile I wrote the second version of the problem without hashing but just an encrypted password. I hoped this makes the analysis harder. But in the same day evening I received an e-mail from Anton Bukov:
Answer for NORCPU hackme challenge, Version 2: “R0und2 D0ne!” ?
It was the correct one. I wondered how is it possible to figure out so quickly?
Anton also shared his approach.
My hack is based on the line:
mem = mem_0.slice(0);
An array gets copied here, and if you call
calc()
function twice on the same array, then for any incorrect password it will print out the answer. In code it looked as:
wcout << calc(L"0") << endl << calc(L"0") << endl;
I guess that copying was used to prevent it. I’m afraid this is not the way you expected. Even I was surprised when got the answer in the output. Afterwards I figured out where it appeared from.
Accidentally Anton discovered a bug causing the program itself to reveal the secret, again. It needed just to run the interpreter twice without resetting memory.
The root of the problem:
... IS_0(flag) JZ okay EXIT(1) okay: ; print the secret ...
After the first wrong run and refusing to print out the secret the program has stopped at the line 4, the ip
register points to the okay
label. If the interpreter gets started again without resetting memory (all registers including ip
reside in memory), it keeps executing right from the okay
and prints out the secret.
It was a shame. I fixed the problem quickly and released the version 2.1 without this side effect.
A couple of days left.
I received an e-mail from a guy with a5b nickname.
Hot-patch of branching and the answer pw:
abcd
resp:R0und2 D0ne!
- the password, obviously, is incorrect
- inverted data being written to
(i == 27692 i==31712)
Formally, this is the solution, but there is no password which means the algorithm is not analysed so far.
But after an hour I received an addendum:
h1cKmE1fUsAn
input data:
chr conI LEET xor
CHR1 13417 13313 104 h
CHR2 39953 39968 49 1
CHR3 54302 54397 99 c
CHR4 32223 32148 75 K
CHR5 30900 30937 109 m
CHR6 27373 27304 69 E
CHR7 16420 16405 49 1
CHR8 49210 49244 102 f
CHR9 16740 16689 85 U
CHR10 50115 50096 115 s
CHR11 19308 19245 65 A
CHR12 57802 57764 110 n
static init:
CHRi 59609= 59651
CONi 59610= 59634
CNTR 59611=12
SUM 59608=0
LEET 59607 = 13313
1:
[59609]+1 -> [59609] // select next chr
[59610]+1 -> [59610] // select next con
SUM |= CHRi ^ LEET ^ CONi
LEET = LEET * 3 + 29
[59611]: if([59611] != 0) Loop 1
...
if(SUM != 0) exit
else print R0und2 D0ne // haven't analysed this part.
// Based on the code modifications (moving the index), it loads 12 constants:
// 29528 22899 2971 9089 27542 17353 52278 25635 11626 34909 39131 51838,
// deals with each one and prints out.
This is the solution now. a5b became the first one who sent the algorithm of the problem 2.
In the same evening I received a solution from Max Filippov.
Algorithm that was used to check password correctness in the first round was the following:
bool check(const char *p) { int v = 0x1040; for(; *p; ++p) { v ^= *p; for (int i = 0; i < 8; ++i) { int f = v & 1; v >>= 1; if (f) v ^= 0x1408; } } return v == 0x1c89; }
that is, sort of CRC.
To discover it I’ve collected NORCPU execution trace and “disassembled” it.
Modified NORCPU source and disassembler are attached, and also may be found there.
And a little add-on:
The method used is pretty straightforward:
- collect execution trace;
- recognize instruction patterns and collapse sequences of primitive instructions to more complex ones;
- analyze disassembled trace.
So, first I needed trace: I copied javascript text into cpp source, fixed lingual differences and inserted the following printf:
while (1) { int i = mem[ip]; printf("%04x:NOR(%04x, %04x => %04x) ", i, mem[i], mem[i + 1], mem[i + 2]); int a = mem[i + 0];
so that I got a long line (about 8Mb) of primitive instruction execution trace.
Then I started constructing sed script that would make it readable.
First, it broke the trace linewise, one instruction per line (288323 lines, will read it in case of insomnia). I took a look at processed trace and recorded several obvious instruction patterns into sed. Then reran script, took next look, recorded more patterns, …
This way I figured out all boolean logic commands and jumps. Then rotations left. Each time new command got recognized, new filtered processed trace was suggesting next step, e.g. 15 ROTL equals ROTR etc.
Then I looked into your article. And found addition pattern in disassembly. And recorded it in sed script.
After that I was able to just read the trace (which shrink to 1035 lines). Its inner loop fit into one page, I just made some notes on a scratchpad:
[f1ba]: current in-T index (i)
[f1b4]: LEN
[f1b5]: 8
0012-0035:[f1b9] ^= (T[i] & 0xff)
006d-007b:[f1b8] = [f1b9] & 1
008a-0158:[f1b9] >>= 1
0167-10c7:[f1aa] = [f1b8] + -1, [f1ab] = !carry
10ca-10e6:jmp on carry to 1145:110d
110d-111b:[f1b9] ^= 1408
1145-1f4f:--[f1b5]
20a5-3005:[f1aa] = [f1b5] + -1, f1ab = !carry
3008-3024:jmp on carry to 006d:304b
304b-304b:++i
3fab-3fab:--LEN
4f0b-5e8a:jmp on carry to 5eb1:6
then I browsed through the repetitions of this inner loop and found the end of the outer loop.
5eb1-6e74:check 1c89
Then just translated it into C. It all took me three evenings.
Then Max Filippov sent the solution for the problem 2.
The second problem answer –
h1cKmE1fUsAn
The result –
R0und2 D0ne!
The password check algorithm is:
bool check(const char *p) { static const int xor_array[] = { 0x3469, 0x9c11, 0xd41e, 0x7ddf, 0x78b4, 0x6aed, 0x4024, 0xc03a, 0x4164, 0xc3c3, 0x4b6c, 0xe1ca, }; int v = 0; int x = 0x3401; for (int i = 0; i < 12; ++i) { int f = p[i] ^ x ^ xor_array[i]; // printf("x: %04x, f: %04x\n", x, f); v |= f; x = (x * 3 + 0x1d) & 0xffff; } return !v; }
The commented printf prints out the secret phrase on-the-fly.
The method of analysis – similar to the first problem – disassembling of the trace.
The first round was frankly more interestintg.
And, finally, the last solution of the first problem is received from Salo Kril.
No comments – just sources.
// Password generation. // Brute_force(3); WORD ks_f(char *buff, int len) { int i, j; WORD ks = 0x1040; for (i = 0; i < len; i++) { ks ^= buff[i]; for (j = 0; j < 8; j++) { if((ks & 1) == 0) ks = ks >> 1; else ks = (ks >> 1) ^ 0x1408; } } return ks; } void Brute_force(int n) { int i; static char alphabet[] = "\x20\x21\x22\x23\x24\x25\x26\x27\x28\x29\x2A\x2B\x2C\x2D\x2E\x2F" "\x30\x31\x32\x33\x34\x35\x36\x37\x38\x39\x3A\x3B\x3C\x3D\x3E\x3F" "\x40\x41\x42\x43\x44\x45\x46\x47\x48\x49\x4A\x4B\x4C\x4D\x4E\x4F" "\x50\x51\x52\x53\x54\x55\x56\x57\x58\x59\x5A\x5B\x5C\x5D\x5E\x5F" "\x60\x61\x62\x63\x64\x65\x66\x67\x68\x69\x6A\x6B\x6C\x6D\x6E\x6F" "\x70\x71\x72\x73\x74\x75\x76\x77\x78\x79\x7A\x7B\x7C\x7D\x7E"; if(n == 0) { if(ks_f(buff_bf, BF_N) == 0x1c89) printf("%s\n",buff_bf); return; } n--; for (i = 0; alphabet[i]; i++) { buff_bf[n] = alphabet[i]; Brute_force(n); } }
and re-constructed code:
#define DEST_COUNT 0xF1FE extern WORD mem[]; /* Secret code: 139471 */ void reconstructed_fn(void) { int i, j; WORD src, dest, key, count, hash, hash_OK, key_const; src = mem[0xF1BA]; // input string count = mem[0xF1ED]; // input string length dest = mem[0xF1BB]; // 0xf1ff hash_OK = mem[0xF1BC]; // 0x1c89 hash = mem[0xF1B9]; // 0x1040 for(i = 0; i < count; i++) { hash ^= mem[src + i] & 0xFF; for (j = 0; j < 8; j++) { if ((hash & 1) == 0) hash = hash >> 1; else hash = (hash >> 1) ^ 0x1408; } } if (hash == hash_OK) { src = mem[0x6EB6]; // "Secret code: 139471" count = mem[0xF1C7]; // 19 key = ((hash >> 8) ^ hash) + 1; key_const = mem[0x6EA8]; // 11 } else { src = mem[0x5EBE]; // "Wrong password!" count = mem[0xF1DC]; // 15 key = mem[0xF1BD]; key_const = mem[0xF1BE]; // 17 } mem[DEST_COUNT] = count; for (i = 0; i < count; i++) { mem[dest + i] = mem[src + i] ^ key; key = key * 3 + key_const; } } /* -------------------------------------------------------------------------------------------- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -------------------------------------------------------------------------------------------- */ WORD and(WORD w1, WORD w2) { return w1 & w2; } WORD or(WORD w1, WORD w2) { return w1 | w2; } WORD xor(WORD w1, WORD w2) { return w1 ^ w2; } WORD rol(WORD w1, WORD n) { return (w1 << n) | (w1 >> (16 - n)); } WORD ror(WORD w1, WORD n) { return (w1 >> n) | (w1 << (16 - n)); } WORD extend_16(WORD k1) { int i, k2; for (i = 0, k2 = 0; i < 16; i++) { k2 = or(k2, k1); k1 = rol(k1, 1); } return k2; } WORD add(WORD *kk1, WORD a, WORD b) { WORD mask_bit, aa0, aa1, tmp1, i, k1, k2; k1 = 0; k2 = 0; mask_bit = 1; for (i = 0; i < 16; i++) { k1 = and(k1, mask_bit); aa0 = xor(a, b); tmp1 = xor(aa0, k1); tmp1 = and(tmp1, mask_bit); k2 = or(tmp1, k2); aa1 = and(a, b); aa0 = and(k1, aa0); k1 = or(aa1, aa0); k1 = rol(k1, 1); mask_bit = rol(mask_bit, 1); } k1 = and(k1, mask_bit); *kk1 = extend_16(k1); return k2; } void reconstr(void) { WORD k1, k2, src, dest, tmp1, key, count, tmp2, tmp3, i, ks, ks_OK, key_a; WORD mask_bit, aa1, aa0; k1 = mem[0xF1AF]; // 0x88 k2 = mem[0xF1A3]; // 0x47 src = mem[0xF1BA]; // input string count = mem[0xF1ED]; // input string length dest = mem[0xF1BB]; // 0xf1ff ks_OK = mem[0xF1BC]; // 0x1c89 "w":ks=0x0ACC "WWW":0x0FCE "123456789":ks=0x05E3 ks = mem[0xF1B9]; // 0x1040 l_0006: ks = xor(ks, and(mem[src], 0xFF)); i = 8; l_006D: tmp3 = and(ks, 1); ks = ror(ks, 1); ks = and(ks, 0x7FFF); tmp2 = add(&k2, tmp3, -1); if(k2 != 0) { ks = xor(ks, 0x1408); } l_1145: i = add(&k1, i, -1); tmp2 = add(&k2, i, -1); if(k2 != 0) goto l_006D; l_304B: src = add(&k1, src, 1); count = add(&k1, count, -1); tmp2 = add(&k2, count, -1); if(k2 != 0) goto l_0006; src = mem[0x5EBE]; // Wrong password! count = mem[0xF1DC]; // 15 key = mem[0xF1BD]; key_a = mem[0xF1BE]; ks_OK = xor(ks_OK, ks); tmp2 = add(&k2, ks_OK, -1); if(k2 != 0) goto l_8552; l_6E9B: src = mem[0x6EB6]; // Secret code: 139471 count = mem[0xF1C7]; // 19 key = ks; key_a = mem[0x6EA8]; key = ror(key, 8); key = and(key, 0xFF); key = xor(ks, key); key = and(key, 0x7FFF); key = add(&k1, key, 1); l_8552: mem[DEST_COUNT] = count; l_8558: mem[dest] = xor(mem[src], key); tmp3 = add(&k1, key, key); key = add(&k1, key, tmp3); key = add(&k1, key, key_a); src = add(&k1, src, 1); dest = add(&k1, dest, 1); count = add(&k1, count, -1); tmp2 = add(&k2, count, -1); if(k2 != 0) goto l_8558; l_F186: mem[0xF1B2] = mem[0xF193]; // nc }
Well, as you understand the solutions have been fully comprehensive.
Thanks to all participants.
I’m putting the original sources below. Just run the Python script to compile the code written on the function-macros, to run through and to generate an html-page (“template.html” is required).
All the sources are available on as a git repository.
File norcpu.py (template.html):
import sys, re, time, string, binascii verbose = False verbose_cpu = False scramble = True test_wrong_crc = 0 secret_code = "Secret code: 139471" password = "h0cKmE1fUsAn" guess = "123456789012" guess = password # Secret code message encryption mask. secret_coef_add = 17 message_text = "Wrong password!" # Wrong password message encryption mask. message_mask = 0x6301 message_coef_add = 11 # Non-standard CRC initial value (should be 0xFFFF). crc16_initial_value = 0x1040 # Non-standard CRC constant (should be 0x8401). crc16_constant = 0x1408 code_segment = [] data_segment = [] label_count = 0 def dump(data, length = 8): result = [] for i in xrange(0, len(data), length): line = data[i:i + length] hex_line = ' '.join(["%04X" % x for x in line]) result.append("%04X: %-*s\n" % (i, length*5, hex_line)) return ''.join(result) def dump_js(data, length = 8): result = [] for i in xrange(0, len(data), length): line = data[i:i + length] hex_line = ' '.join(["0x%04X," % x for x in line]) result.append("%-*s\n" % (length*5, hex_line)) return ''.join(result) def calc_crc16(data): global crc16_initial_value global crc16_constant crc16 = crc16_initial_value for i in range(0, len(data)): ch = ord(data[i]) & 0xff crc16 = crc16 ^ ch for j in range(0, 8): if ((crc16 & 1) != 0): crc16 = (crc16 >> 1) ^ crc16_constant else: crc16 = crc16 >> 1 return crc16 crc16 = calc_crc16(password) def encode_string(data, name, mask, coef_add): global mem, names offset = names[name] offset_sz = names[name + "_sz"] for i in range(0, len(data)): mem[offset + i] = ord(data[i]) ^ mask mask = (mask * 3 + coef_add) & 0xffff mem[offset_sz] = len(data) def put_string(data, name): global mem, names offset = names[name] offset_sz = names[name + "_sz"] for i in range(0, len(data)): mem[offset + i] = ord(data[i]) mem[offset_sz] = len(data) def save_mem(name, size = -1): f = open(name, "w") if size == -1: size = len(mem) for i in (mem[0:size]): hex = "%04X" % i bin = binascii.a2b_hex(hex) f.write(bin) f.close() def next_label(): global label_count label_count = label_count + 1 return "label_%04d" % label_count def code_rem(comment): code_segment.append('; ' + comment) def data_rem(comment): data_segment.append('; ' + comment) def data_label(name): data_segment.append(name + ":") def code_label(name): code_segment.append(name + ":") def code(value): printed = value if type(value).__name__ == 'int': printed = "%d" % value code_segment.append(" dw %s" % printed) scramble_counter = 0x27 def next_scramble_counter(): global scramble_counter scramble_counter = scramble_counter * 3 + 7 return scramble_counter & 0xff def word(value): if value == -1: if scramble: value = next_scramble_counter() else: value = 0 printed = value if type(value).__name__ == 'int': printed = "%d" % value data_segment.append(" dw %s" % printed) def buffer(length, value = -1): for i in range(0, length): word(value) def var(name, value = -1): data_label(name); word(value); def NOR(a, b, r): code_rem('NOR ' + str(a) + ' ' + str(b) + ' ' + str(r)) code(a) code(b) code(r) def NOT(a, r): NOR(a, a, r); def OR(a, b, r): NOR(a, b, "or_reg") NOT("or_reg", r) var("or_reg") def AND(a, b, r): NOT(a, "and_reg_a") NOT(b, "and_reg_b") OR("and_reg_a", "and_reg_b", "and_reg_a") NOT("and_reg_a", r) var("and_reg_a") var("and_reg_b") def ANDi(a, imm, r): MOVi(imm, "and_i_reg") AND(a, "and_i_reg", r) var("and_i_reg") def XOR(a, b, r): NOT(a, "xor_reg_a") NOT(b, "xor_reg_b") AND(a, "xor_reg_b", "xor_reg_b") AND(b, "xor_reg_a", "xor_reg_a") OR("xor_reg_a", "xor_reg_b", r) var("xor_reg_a") var("xor_reg_b") def XORi(a, imm, r): MOVi(imm, "xor_i_reg") XOR(a, "xor_i_reg", r) var("xor_i_reg") def MOV(a, b): code_rem('MOV ' + str(a) + ' ' + str(b)) NOT(a, "move_reg") NOT("move_reg", b) code_rem('MOV END') var("move_reg") def JMP(a): code_rem('JMP ' + str(a)) MOV(a, "ip") def JMPi(a): code_rem('JMPi ' + str(a)) label = next_label() JMP(label) code_label(label) code(a) def MOVi(imm, a): code_rem('MOVi #' + str(imm) + ' ' + str(a)) label_data = next_label() label_jump = next_label() MOV(label_data, a) JMPi(label_jump) code_label(label_data) code(imm) code_label(label_jump) # [a] -> b def PEEK(a, b): label1 = next_label() label2 = next_label() MOV(a, label1) MOV(a, label2) code_label(label1) # NOT(0, 0, move_reg) code(0) # <- a code_label(label2) # code(0) # <- a code("move_reg") # NOT("move_reg", b) # a -> [b] def POKE(a, b): code_rem('POKE ' + str(a) + ' [' + str(b) + ']') label = next_label() MOV(b, label) NOT(a, "move_reg") # +3 (three operations) code("move_reg") # +4 code("move_reg") # +5 code_label(label) code(0) # <- b # imm -> [a] def POKEi(imm, a): MOVi(imm, "poke_i_reg") POKE("poke_i_reg", a) var("poke_i_reg") def EXIT(a): MOV(a, "exit_reg") def EXITi(a): MOVi(a, "exit_reg") def FADD(mask, carry, a, b, r): AND(a, mask, "fadd_reg_a") # zero bits in 'a' except mask'ed AND(b, mask, "fadd_reg_b") # zero bits in 'b' except mask'ed AND(carry, mask, carry) # zero bits in 'carry' except mask'ed # SUM = (a ^ b) ^ carry XOR(a, b, "fadd_reg_t1") XOR("fadd_reg_t1", carry, "fadd_reg_bit_r") # Leave only 'mask'ed bit in bit_r. AND("fadd_reg_bit_r", mask, "fadd_reg_bit_r") # Add current added bit to the result. OR("fadd_reg_bit_r", r, r) # CARRY = (a & b) | (carry & (a ^ b)) AND(a, b, "fadd_reg_t2") AND(carry, "fadd_reg_t1", "fadd_reg_t1") # CARRY is calculated, and 'shift_reg' contains the same value # but shifted the left by 1 bit. OR("fadd_reg_t2", "fadd_reg_t1", carry) # CARRY is shifted the left by 1 bit to be used on the next round. MOV("shift_reg", carry) # shift_reg = mask << 1 MOV(mask, mask) # mask = shift (effectively "mask = mask << 1") MOV("shift_reg", mask) AND(carry, mask, carry) var("fadd_reg_a") var("fadd_reg_b") var("fadd_reg_bit_r") var("fadd_reg_t1") var("fadd_reg_t2") def ZERO(a): XOR(a, a, a) def FADC(a, b, r): ZERO("fadc_reg_t") MOV("const_1", "fadc_reg_mask") for i in range(0, 16): FADD("fadc_reg_mask", "carry_reg", a, b, "fadc_reg_t") MOV("fadc_reg_t", r) ZERO("fadc_reg_t") for i in range(0, 16): OR("fadc_reg_t", "carry_reg", "fadc_reg_t") MOV("carry_reg", "carry_reg") MOV("shift_reg", "carry_reg") MOV("fadc_reg_t", "carry_reg") var("fadc_reg_mask") var("fadc_reg_t") def ADD(a, b, r): ZERO("carry_reg") FADC(a, b, r) def ADDi(a, imm, r): MOVi(imm, "add_i_reg") ADD(a, "add_i_reg", r) var("add_i_reg") def PUSH(a): ADD("stack_reg", "const_minus_1", "stack_reg") POKE(a, "stack_reg") def PUSHi(imm): MOVi(imm, "push_i_reg") PUSH("push_i_reg") var("push_i_reg") def POP(a): PEEK("stack_reg", a) ADD("stack_reg", "const_1", "stack_reg") def CALL(a): label = next_label() PUSHi(label) JMP(a) code_label(label) def CALLi(a): label = next_label() PUSHi(label) JMPi(a) code_label(label) def RET(): POP("ip") # Jump 'a', if cond = FFFF, and 'b' if conf = 0000 def BRANCH(a, b, cond): AND(a, cond, "branch_reg_a") # reg_a = a & cond NOT(cond, "branch_reg_b") # reg_b = !cond AND(b, "branch_reg_b", "branch_reg_b") # reg_b = b & reg_b = b & !cond OR("branch_reg_a", "branch_reg_b", "ip") # ip = (a & cond) | (b & !cond) var("branch_reg_a") var("branch_reg_b") # Jump 'a', if cond = FFFF, and 'b' if conf = 0000 def BRANCHi(a, b, cond): MOVi(a, "branch_i_reg_a") MOVi(b, "branch_i_reg_b") BRANCH("branch_i_reg_a", "branch_i_reg_b", cond) var("branch_i_reg_a") var("branch_i_reg_b") # if a != 0 -> carry = FFFF else carry = 0000 def IS_0(a): ZERO("carry_reg") FADC(a, "const_minus_1", "is_0_reg") NOT("carry_reg", "zero_reg") var("is_0_reg") var("zero_reg") # ip = (zero_reg == FFFF ? a : ip) def JZi(a): label = next_label() BRANCHi(a, label, "zero_reg") code_label(label) # ip = (zero_reg == FFFF ? a : ip) def JNZi(a): label = next_label() BRANCHi(label, a, "zero_reg") code_label(label) def ROL(a, b): MOV(a, a) # shift_reg = a << 1 MOV("shift_reg", b) def ROR(a, b): MOV(a, "ror_reg") for i in range(0, 15): ROL("ror_reg", "ror_reg") MOV("ror_reg", b) var("ror_reg") def SHL(a, b): ROL(a, b) ANDi(b, 0x0001, b) def SHR(a, b): ROR(a, b) ANDi(b, 0x7FFF, b) # NORCPU code var("ip", "start") var("shift_reg") var("carry_reg") var("const_1", 1) var("const_minus_1", 0xFFFF) var("exit_reg") var("stack_reg", "stack") code_label("start") var("i") var("j") var("ch") var("mask") var("t") var("crc16", crc16_initial_value) var("ptr", "password") MOV("password_sz", "i") crc_loop = next_label() code_label(crc_loop) # crc_loop # ch = *ptr PEEK("ptr", "ch") # ch &= 0xFF ANDi("ch", 0xFF, "ch") # crc16 ^= ch XOR("crc16", "ch", "crc16") MOVi(8, "j") crc_loop_j = next_label() code_label(crc_loop_j) # crc_loop_j # t = crc16 & 1 ANDi("crc16", 1, "t") # crc16 >>= 1 SHR("crc16", "crc16") IS_0("t") crc_loop_1 = next_label() JZi(crc_loop_1) # crc16 ^= crc16_constant XORi("crc16", crc16_constant, "crc16") code_label(crc_loop_1) # crc_loop_1 ADD("j", "const_minus_1", "j") IS_0("j") JNZi(crc_loop_j) # ptr += 1 ADD("ptr", "const_1", "ptr") # i = i - 1 ADD("i", "const_minus_1", "i") IS_0("i") JNZi(crc_loop) var("ptr2", "result") correct_crc = crc16 + test_wrong_crc var("correct_crc", correct_crc) # By default we're going to decrypt 'Wrong...' message. MOVi("message", "ptr") MOV("message_sz", "i") var("message_mask", message_mask) MOV("message_mask", "mask") var("coef_add", message_coef_add) wrong_label = next_label() XOR("correct_crc", "crc16", "correct_crc") IS_0("correct_crc") JNZi(wrong_label) # Now we switch to descrypt the secret message. MOVi(secret_coef_add, "coef_add") MOVi("secret", "ptr") MOV("secret_sz", "i") # mask = ((crc16 & 0xff) | ((crc16 >> 8) & 0xff)) + 1 MOV("crc16", "mask") for i in range(0, 8): SHR("mask", "mask") XOR("crc16", "mask", "mask") ANDi("mask", 0xff, "mask") ADD("mask", "const_1", "mask") code_label(wrong_label) MOV("i", "result_sz") loop = next_label() code_label(loop) # loop # ch = *ptr PEEK("ptr", "ch") # ch ^= mask XOR("ch", "mask", "ch") POKE("ch", "ptr2") # mask = mask * 3 + 11 ADD("mask", "mask", "t") ADD("mask", "t", "mask") ADD("mask", "coef_add", "mask") # ptr += 1 ADD("ptr", "const_1", "ptr") # ptr2 += 1 ADD("ptr2", "const_1", "ptr2") # i = i - 1 ADD("i", "const_minus_1", "i") IS_0("i") JNZi(loop) EXITi(0x00) buffer(8) data_label("stack") var("secret_sz", len(secret_code)) data_label("secret") buffer(len(secret_code) + 1) var("message_sz") data_label("message") buffer(16) var("password_sz") data_label("password") buffer(16) # The buffer holding the result string. var("result_sz") data_label("result") buffer(32) # Compiler text = code_segment text.extend(data_segment) if verbose: print "\n".join(text) # Phase 1. Calculate names. addr = 0 names = {} for line in text: if line[0] == ';': continue if line[0] != ' ': name = line.partition(':')[0] names[name] = addr else: addr = addr + 1 if verbose: print names raw_text = "\n".join(text) # Resolve names. for name in names: if verbose: print name, names[name], type(names[name]) name_re = re.compile(r'dw ' + name + '$', re.M) value = "%d" % names[name] raw_text = name_re.sub('dw ' + value, raw_text) text = raw_text.split("\n") if verbose: print "\n".join(text) # Phase 2. Compilation. addr = 0 comment = "" mem = [] for line in text: if line[0] == ';' or line[0] != ' ': comment = comment + line + ' ' else: value = int(line.strip().partition(" ")[2]) if verbose: print "%04X: %04X ; %s" % (addr, value, comment) mem.append(value) addr = addr + 1 comment = "" # Interpretation ip = names["ip"] exit_reg = names["exit_reg"] shift_reg = names["shift_reg"] carry_reg = names["carry_reg"] def nor(a, b): r = a | b r = r ^ 0xFFFF return r & 0xFFFF def norcpu(): while 1: i = mem[ip]; a = mem[i + 0] b = mem[i + 1] r = mem[i + 2] mem[ip] = i + 3 f = nor(mem[a], mem[b]) mem[r] = f mem[shift_reg] = ((f >> 15) & 1) | ((f & 0x7FFF) << 1) if verbose_cpu: print "%04X: %04X [%04X] %04X [%04X] -> %04X [%04X]" % \ (i, a, mem[a], b, mem[b], r, mem[r]) if r == exit_reg: break print "Starting from [%04X]" % mem[ip] # Encrypt the secret code. secret_mask = ((crc16 & 0xff) ^ ((crc16 >> 8) & 0xff)) + 1 encode_string(secret_code, "secret", secret_mask, secret_coef_add); # Encrypt 'Wrong...' message. encode_string(message_text, "message", message_mask, message_coef_add); mem_js = dump_js(mem) save_mem("norcpu-1-before.bin") mem_sz = len(mem) if len(mem) >= 0x10000: print "Too much code (%08X, %04X)" % (len(mem), len(mem) - 0x10000) sys.exit() # Inject plain password in the last moment (for testing). put_string(guess, "password") save_mem("norcpu-2-before-with-password.bin") if verbose: print "Original memory:" print dump(mem) start_time = time.time() norcpu() end_time = time.time() save_mem("norcpu-3-after.bin", mem_sz) if verbose: print "Memory after:" dump(mem) print print "Size: %X" % len(mem) print "Time: %d" % (end_time - start_time) print "Exit: %04X" % mem[exit_reg] print("CRC : %04X (%04X)" % (crc16, correct_crc)) result = names["result"] result_value = "" for i in range(0, mem[names["result_sz"]]): result_value = result_value + chr(mem[result + i] & 0xff) if result_value != secret_code: print "ERROR: [%s] != [%s]" % (secret_code, result_value) js = string.Template(open('template.html', 'r').read()) js = js.substitute( \ ip = names["ip"], exit_reg = names["exit_reg"], shift_reg = names["shift_reg"], password = names["password"], password_sz = names["password_sz"], result = names["result"], result_sz = names["result_sz"], mem_js = mem_js ) f = open("norcpu.html", "w") f.write(js) f.close()
File norcpu.py (template.html):
import sys, re, time, string, binascii verbose = False verbose_cpu = False scramble = True secret_password = "h1cKmE1fUsAn" secret_password_xor_mask = 0x3401 secret_password_add = 29 secret_code = "R0und2 D0ne!" secret_code_xor_mask = 0x730A secret_code_add = 37 guess = "123456789012" guess = secret_password code_segment = [] data_segment = [] label_count = 0 def dump(data, length = 8): result = [] for i in xrange(0, len(data), length): line = data[i:i + length] hex_line = ' '.join(["%04X" % x for x in line]) result.append("%04X: %-*s\n" % (i, length*5, hex_line)) return ''.join(result) def dump_js(data, length = 8): result = [] for i in xrange(0, len(data), length): line = data[i:i + length] hex_line = ' '.join(["0x%04X," % x for x in line]) result.append("%-*s\n" % (length*5, hex_line)) return ''.join(result) def encode_string(data, name, mask, coef_add): global mem, names offset = names[name] offset_sz = names[name + "_sz"] for i in range(0, len(data)): mem[offset + i] = ord(data[i]) ^ mask mask = (mask * 3 + coef_add) & 0xffff mem[offset_sz] = len(data) def put_string(data, name): global mem, names offset = names[name] offset_sz = names[name + "_sz"] for i in range(0, len(data)): mem[offset + i] = ord(data[i]) mem[offset_sz] = len(data) def save_mem(name, size = -1): f = open(name, "w") if size == -1: size = len(mem) for i in (mem[0:size]): hex = "%04X" % i bin = binascii.a2b_hex(hex) f.write(bin) f.close() def next_label(): global label_count label_count = label_count + 1 return "label_%04d" % label_count def code_rem(comment): code_segment.append('; ' + comment) def data_rem(comment): data_segment.append('; ' + comment) def data_label(name): data_segment.append(name + ":") def code_label(name): code_segment.append(name + ":") def code(value): printed = value if type(value).__name__ == 'int': printed = "%d" % value code_segment.append(" dw %s" % printed) scramble_counter = 0x2743 def next_scramble_counter(): global scramble_counter scramble_counter = scramble_counter * 3 + 7 return scramble_counter & 0xffff def word(value): if value == -1: if scramble: value = next_scramble_counter() else: value = 0 printed = value if type(value).__name__ == 'int': printed = "%d" % value data_segment.append(" dw %s" % printed) def buffer(length, value = -1): for i in range(0, length): word(value) def var(name, value = -1): data_label(name); word(value); # Macros def NOR(a, b, r): code_rem('NOR ' + str(a) + ' ' + str(b) + ' ' + str(r)) code(a) code(b) code(r) def NOT(a, r): NOR(a, a, r); def OR(a, b, r): NOR(a, b, "or_reg") NOT("or_reg", r) var("or_reg") def AND(a, b, r): NOT(a, "and_reg_a") NOT(b, "and_reg_b") OR("and_reg_a", "and_reg_b", "and_reg_a") NOT("and_reg_a", r) var("and_reg_a") var("and_reg_b") def ANDi(a, imm, r): MOVi(imm, "and_i_reg") AND(a, "and_i_reg", r) var("and_i_reg") def XOR(a, b, r): NOT(a, "xor_reg_a") NOT(b, "xor_reg_b") AND(a, "xor_reg_b", "xor_reg_b") AND(b, "xor_reg_a", "xor_reg_a") OR("xor_reg_a", "xor_reg_b", r) var("xor_reg_a") var("xor_reg_b") def XORi(a, imm, r): MOVi(imm, "xor_i_reg") XOR(a, "xor_i_reg", r) var("xor_i_reg") def MOV(a, b): code_rem('MOV ' + str(a) + ' ' + str(b)) NOT(a, "move_reg") NOT("move_reg", b) code_rem('MOV END') var("move_reg") def JMP(a): code_rem('JMP ' + str(a)) MOV(a, "ip") def JMPi(a): code_rem('JMPi ' + str(a)) label = next_label() JMP(label) code_label(label) code(a) def MOVi(imm, a): code_rem('MOVi #' + str(imm) + ' ' + str(a)) label_data = next_label() label_jump = next_label() MOV(label_data, a) JMPi(label_jump) code_label(label_data) code(imm) code_label(label_jump) # [a] -> b def PEEK(a, b): label1 = next_label() label2 = next_label() MOV(a, label1) MOV(a, label2) code_label(label1) # NOT(0, 0, move_reg) code(0) # <- a code_label(label2) # code(0) # <- a code("move_reg") # NOT("move_reg", b) # a -> [b] def POKE(a, b): code_rem('POKE ' + str(a) + ' [' + str(b) + ']') label = next_label() MOV(b, label) NOT(a, "move_reg") # +3 (three operations) code("move_reg") # +4 code("move_reg") # +5 code_label(label) code(0) # <- b # imm -> [a] def POKEi(imm, a): MOVi(imm, "poke_i_reg") POKE("poke_i_reg", a) var("poke_i_reg") def EXIT(a): MOV(a, "exit_reg") def EXITi(a): MOVi(a, "exit_reg") def FADD(mask, carry, a, b, r): AND(a, mask, "fadd_reg_a") # zero bits in 'a' except mask'ed AND(b, mask, "fadd_reg_b") # zero bits in 'b' except mask'ed AND(carry, mask, carry) # zero bits in 'carry' except mask'ed # SUM = (a ^ b) ^ carry XOR(a, b, "fadd_reg_t1") XOR("fadd_reg_t1", carry, "fadd_reg_bit_r") # Leave only 'mask'ed bit in bit_r. AND("fadd_reg_bit_r", mask, "fadd_reg_bit_r") # Add current added bit to the result. OR("fadd_reg_bit_r", r, r) # CARRY = (a & b) | (carry & (a ^ b)) AND(a, b, "fadd_reg_t2") AND(carry, "fadd_reg_t1", "fadd_reg_t1") # CARRY is calculated, and 'shift_reg' contains the same value # but shifted the left by 1 bit. OR("fadd_reg_t2", "fadd_reg_t1", carry) # CARRY is shifted the left by 1 bit to be used on the next round. MOV("shift_reg", carry) # shift_reg = mask << 1 MOV(mask, mask) # mask = shift (effectively "mask = mask << 1") MOV("shift_reg", mask) AND(carry, mask, carry) var("fadd_reg_a") var("fadd_reg_b") var("fadd_reg_bit_r") var("fadd_reg_t1") var("fadd_reg_t2") def ZERO(a): XOR(a, a, a) def FADC(a, b, r): ZERO("fadc_reg_t") MOV("const_1", "fadc_reg_mask") for i in range(0, 16): FADD("fadc_reg_mask", "carry_reg", a, b, "fadc_reg_t") MOV("fadc_reg_t", r) ZERO("fadc_reg_t") for i in range(0, 16): OR("fadc_reg_t", "carry_reg", "fadc_reg_t") MOV("carry_reg", "carry_reg") MOV("shift_reg", "carry_reg") MOV("fadc_reg_t", "carry_reg") var("fadc_reg_mask") var("fadc_reg_t") def ADD(a, b, r): ZERO("carry_reg") FADC(a, b, r) def ADDi(a, imm, r): MOVi(imm, "add_i_reg") ADD(a, "add_i_reg", r) var("add_i_reg") def PUSH(a): ADD("stack_reg", "const_minus_1", "stack_reg") POKE(a, "stack_reg") def PUSHi(imm): MOVi(imm, "push_i_reg") PUSH("push_i_reg") var("push_i_reg") def POP(a): PEEK("stack_reg", a) ADD("stack_reg", "const_1", "stack_reg") def CALL(a): label = next_label() PUSHi(label) JMP(a) code_label(label) def CALLi(a): label = next_label() PUSHi(label) JMPi(a) code_label(label) def RET(): POP("ip") # Jump 'a', if cond = FFFF, and 'b' if conf = 0000 def BRANCH(a, b, cond): AND(a, cond, "branch_reg_a") # reg_a = a & cond NOT(cond, "branch_reg_b") # reg_b = !cond AND(b, "branch_reg_b", "branch_reg_b") # reg_b = b & reg_b = b & !cond OR("branch_reg_a", "branch_reg_b", "ip") # ip = (a & cond) | (b & !cond) var("branch_reg_a") var("branch_reg_b") # Jump 'a', if cond = FFFF, and 'b' if conf = 0000 def BRANCHi(a, b, cond): MOVi(a, "branch_i_reg_a") MOVi(b, "branch_i_reg_b") BRANCH("branch_i_reg_a", "branch_i_reg_b", cond) var("branch_i_reg_a") var("branch_i_reg_b") # if a != 0 -> carry = FFFF else carry = 0000 def IS_0(a): ZERO("carry_reg") FADC(a, "const_minus_1", "is_0_reg") NOT("carry_reg", "zero_reg") var("is_0_reg") var("zero_reg") # ip = (zero_reg == FFFF ? a : ip) def JZi(a): label = next_label() BRANCHi(a, label, "zero_reg") code_label(label) # ip = (zero_reg == FFFF ? a : ip) def JNZi(a): label = next_label() BRANCHi(label, a, "zero_reg") code_label(label) def ROL(a, b): MOV(a, a) # shift_reg = a << 1 MOV("shift_reg", b) def ROR(a, b): MOV(a, "ror_reg") for i in range(0, 15): ROL("ror_reg", "ror_reg") MOV("ror_reg", b) var("ror_reg") def SHL(a, b): ROL(a, b) ANDi(b, 0x0001, b) def SHR(a, b): ROR(a, b) ANDi(b, 0x7FFF, b) def MUL3(a, b): ADD(a, a, "mul3_reg") # mul3_reg = a + a ADD("mul3_reg", a, b) # b = mul3_reg + a var("mul3_reg") # NORCPU code var("ip", "start") var("shift_reg") var("carry_reg") var("const_1", 1) var("const_minus_1", 0xFFFF) var("exit_reg") var("stack_reg", "stack") code_label("start") var("ch") var("t") var("xor_mask") var("cmp_flag") var("ptr") var("ptr2") var("i") MOVi("exchange", "ptr") MOVi("secret_password", "ptr2") MOVi(secret_password_xor_mask, "xor_mask") MOVi(0, "cmp_flag") MOVi(len(secret_password), "i") cmp_loop = next_label() code_label(cmp_loop) # cmp_loop: PEEK("ptr", "ch") # ch = *ptr XOR("ch", "xor_mask", "ch") # ch ^= xor_mask PEEK("ptr2", "t") # t = *ptr2 XOR("ch", "t", "ch") # ch = ch ^ t OR("cmp_flag", "ch", "cmp_flag") # cmp_flag |= ch ADD("ptr", "const_1", "ptr") # ptr += 1 ADD("ptr2", "const_1", "ptr2") # ptr2 += 1 MUL3("xor_mask", "xor_mask") # xor_mask *= 3 ADDi("xor_mask", secret_password_add, "xor_mask") # xor_mask += add_const ADD("i", "const_minus_1", "i") # i -= 1 IS_0("i") JNZi(cmp_loop) MOVi(0, "exchange_sz") ok_label = next_label() IS_0("cmp_flag") JZi(ok_label) exit_label = next_label() JMPi(exit_label) code_label(ok_label) MOVi("secret_code", "ptr") MOV("secret_code_sz", "i") MOVi(secret_code_xor_mask, "xor_mask") MOVi("exchange", "ptr2") MOV("i", "exchange_sz") loop = next_label() code_label(loop) # loop: PEEK("ptr", "ch") # ch = *ptr XOR("ch", "xor_mask", "ch") # ch ^= xor_mask POKE("ch", "ptr2") # *ptr2 = ch MUL3("xor_mask", "xor_mask") # xor_mask *= 3 ADDi("xor_mask", secret_code_add, "xor_mask") # xor_mask += add_const ADD("ptr", "const_1", "ptr") # ptr += 1 ADD("ptr2", "const_1", "ptr2") # ptr2 += 1 ADD("i", "const_minus_1", "i") # i = i - 1 IS_0("i") JNZi(loop) code_label(exit_label) # exit_label: EXITi(0x00) buffer(8) data_label("stack") var("secret_code_sz", len(secret_code)) data_label("secret_code") buffer(len(secret_code)) var("secret_password_sz") data_label("secret_password") buffer(16) var("exchange_sz", 0) data_label("exchange") buffer(32) # Compiler text = code_segment text.extend(data_segment) if verbose: print "\n".join(text) # Phase 1. Calculate names. addr = 0 names = {} for line in text: if line[0] == ';': continue if line[0] != ' ': name = line.partition(':')[0] names[name] = addr else: addr = addr + 1 if verbose: print names raw_text = "\n".join(text) # Resolve names. for name in names: if verbose: print name, names[name], type(names[name]) name_re = re.compile(r'dw ' + name + '$', re.M) value = "%d" % names[name] raw_text = name_re.sub('dw ' + value, raw_text) text = raw_text.split("\n") if verbose: print "\n".join(text) # Phase 2. Compilation. addr = 0 comment = "" mem = [] for line in text: if line[0] == ';' or line[0] != ' ': comment = comment + line + ' ' else: value = int(line.strip().partition(" ")[2]) if verbose: print "%04X: %04X ; %s" % (addr, value, comment) mem.append(value) addr = addr + 1 comment = "" # Interpretation ip = names["ip"] exit_reg = names["exit_reg"] shift_reg = names["shift_reg"] carry_reg = names["carry_reg"] def nor(a, b): r = a | b r = r ^ 0xFFFF return r & 0xFFFF def norcpu(): while 1: i = mem[ip]; a = mem[i + 0] b = mem[i + 1] r = mem[i + 2] mem[ip] = i + 3 f = nor(mem[a], mem[b]) mem[r] = f mem[shift_reg] = ((f >> 15) & 1) | ((f & 0x7FFF) << 1) if verbose_cpu: print "%04X: %04X [%04X] %04X [%04X] -> %04X [%04X]" % \ (i, a, mem[a], b, mem[b], r, mem[r]) if r == exit_reg: break print "Starting from [%04X]" % mem[ip] encode_string(secret_code, "secret_code", secret_code_xor_mask, secret_code_add); encode_string(secret_password, "secret_password", secret_password_xor_mask, secret_password_add); mem_js = dump_js(mem) save_mem("norcpu-1-before.bin") mem_sz = len(mem) if len(mem) >= 0x10000: print "Too much code (%08X, %04X)" % (len(mem), len(mem) - 0x10000) sys.exit() # Inject plain password in the last moment (for testing). put_string(guess, "exchange") save_mem("norcpu-2-before-with-password.bin") if verbose: print "Original memory:" print dump(mem) start_time = time.time() norcpu() end_time = time.time() save_mem("norcpu-3-after.bin", mem_sz) if verbose: print "Memory after:" dump(mem) print print "Size: %X" % len(mem) print "Time: %d" % (end_time - start_time) print "Exit: %04X" % mem[exit_reg] exchange = names["exchange"] result_value = "" for i in range(0, mem[names["exchange_sz"]]): result_value = result_value + chr(mem[exchange + i] & 0xff) print "Result: [%s]" % result_value if len(result_value) == 0: print "ERROR: Wrong password" js = string.Template(open('template.html', 'r').read()) js = js.substitute( \ ip = names["ip"], exit_reg = names["exit_reg"], shift_reg = names["shift_reg"], exchange = names["exchange"], exchange_sz = names["exchange_sz"], mem_js = mem_js ) f = open("norcpu2.html", "w") f.write(js) f.close()
In the environment I’m describing here you can easily write code manually on the high level macro assembler.
Okay, here is a virtual processor understanding only one logical command - Peirce arrow.
A program written for this processor expects a password as an input. If the password is incorrect, it print out “Wrong password!”. Otherwise the special magic message will be printed.
The problem statement: You have to find out that magic message using any approach. For example, you can guess the password, the program will just print the secret out.
The logic is written such way that knowing the algorithm allows restoring the magic message without any problems.
Some time ago I described this approach in all details.
The original approach I used in this experiment, wasn’t 100% “clear”, because the addition command was implemented outside the processor. In this implementation everything works inside the processor. It required modifying the interpreter slightly introducing a shift register.
For those who want to try to hack this problem, I’ve created a web page containing an implementation of this one-command processor and the programming check the password.
Here you go - the challenge.
P.S. The first one solved the problem will get a little prize! Information is on the page.
I’ve discovered this way of getting technical information for myself only recently. Impression is only positive so far - the perceptual speed is much faster comparing to reading.
For example, after Coding in Objective-C 2.0 I can write a program in this language using classes and have an idea how the memory management works in Objective-C (by the way, it’s quite interesting concept - a “manual” garbage collector when memory is freed automatically but the programmer has to manage the counter of active links to an object by himself.
Currently watching Writing Your First iPhone Application and the next is Erlang in Practice.
This is a delicate moment with screencasts - it should be made qualitatively. Otherwise it’s waste of time.
Which screencasts do you watch? What can you recommend?
P.S. By the way, I like The pragmatic Bookshelf online store more and more. All books are in multiple DRM free formats (pdf, mobi, epub).
1. Pure C++ method based on the references.
class Foo { Value field_; public: Value& field() { return field_; } const Value& field() const { return field_; } };
Usage:
Foo foo; foo.field() = field_instance; field_instance = foo.field();
Pros: brevity, closeness to the property notation and possibility of using in a cascade assignment (foo1.field() = foo2.field() = 2;
).
Cons: using the function call on the left looks unusual.
2. Java way
class Foo { Value field_; public: void setField(const Value& value) { field_ = value; } const Value& getField() const { return field_; } };
Usage:
Foo foo; foo.setField(field_instance); field_instance = foo.getField();
Pros: clarity and obviousness.
Cons: wordiness due to get
and set
prefixes.
3. Objective-C way
class Foo { Value field_; public: void setField(const Value& value) { field_ = value; } const Value& field() const { return field_; } };
Usage:
Foo foo; foo.setField(field_instance); field_instance = foo.field();
Pros: brevity (no useless get
prefix) and clarify.
Cons: haven’t found so far.
All three have rights to live but from the style perspective it’s good to be consistent and use only one style across one project.
115
.
#include <iostream> #include <cmath> using namespace std; int main(int argc, char* argv[]) { double f = 1.15; int a = f * 100.0 + 0.1E-9; int b = f * 100.0; cout << "a = " << a << endl; cout << "b = " << b << endl; return 0; }
No, on VS2008 it prints out:
115
114
Be careful.
In the mid-nineties in the FIDO discussion group RU.HACKER I come across an interesting hackme.
Usually “hackme” is an application which is published as a challenge to be hacked. It prompts for a password and the goal is to crack it. You may guess the password disassembling the code or could just patch the code and disable the password check and so on.
That “hackme” from RU.HACKER was originally written by famous Russian hacker and security expert Alexander Peslyak (the author of John The Ripper John The Ripper and Openwall Linux).
Unlike other “hackme” around Alexander’s approach didn’t have any anti-debugger tricks at all preventing disassembling and tracing. Furthermore it contained less then hundred bytes of x86 code. The code executed a trivial loop calculating only one operation - NOR (Pierce arrow). And the entire business logic (interaction with the console and password calculations) was build on top this primitive. Effectively this is a model of CPU having only one command.
The memory model of this CPU is flat and it consists of 16-bit words. The address space is from 0x0000 to 0xFFFF. A CPU instruction has 3 operands. There is no an instruction code because the command is always the same - NOR. Each operand represents an address.
To execute an instruction CPU takes the left argument of NOR from an address given in the first operand and the right argument from an address given in the second operand. Then it implies bitwise 16-bit NOR between them and puts the result to the address given in the third argument. Afterwards the instruction pointer is increment by 3 to shift to the next instruction and the cycle repeats.
The instruction pointer (IP
) is also located in the address space and JUMP operation could be implemented just by putting a jump address to the memory location of IP
.
The original NOR interpreter was written on x86 assembly:
cld emCPU: mov si,emIP lodsw xchg ax,di mov di,[di] lodsw xchg ax,bx or di,[bx] lodsw xchg ax,di not ax mov emIP,si stosw jmp short emCPU
Nothing prevents to write it on any language, for example, Python.
def nor(a, b): return ~(a | b) & 0xFFFF def norcpu(): while 1: i = mem[IP]; a = mem[i + 0] b = mem[i + 1] r = mem[i + 2] mem[IP] = i + 3 f = nor(mem[a], mem[b]) mem[r] = f
Why NOR? The Boolean algebra theory says that any of 14 of 16 Boolean functions could be calculated using just the rest two - NOR and NAND. For instance:
NOT(a) = NOR(a, a)
AND(a, b) = NOT(OR(NOT(a), NOT(b)))
OR(a, b) = NOT(NOR(a, b))
XOR(a, b) = OR(AND(a, NOT(b)), AND(NOT(a), b)))
The move operation MOVE(src, dst)
could be implemented via OR
:
mem[dst] = OR(mem[src], mem[src])
A conditional jump is either implemented via Boolean functions. If cond
equals 0xFFFF (true), the jump to addr
is performed. If cond
equals 0x0000 (false) CPU sequentially takes the next instruction following JUMP
.
mem[IP] = OR(AND(addr, cond), AND(mem[IP], cond))
Or in the NOR interpreter notation:
AND addr, cond, @t1
AND IP, cond, @t2
OR @t1, @t2, IP
where @t1
and @t2
are temporary variables. The AND
and OR
commands will be also expanded to sets of NORs as shown above.
Eventually we have Boolean functions, copy/move, unconditional and conditional jumps. We just lack the addition/subtraction and shifts. Having that we could implement the subroutine stack and then any complex computations can be made by this primitive CPU.
The original Alexander’s NOR CPU implementation had a workaround which allows running the native x86 code. The workaround was based on the fact that the NOR interpreter also resides in its address space and can modify itself. To execute the native code a special high level macro (let’s say emCallX86) uses MOVE operation (see above) to place two bytes at the beginning of the interpreter code (effectively it saves the original two bytes and put other two instead). Those two new bytes are an x86 short jump instruction. When the interpreter begins the next cycle it simply jumps out to an arbitrary x86 code. In the end the native code restores the original two bytes of the interpreter and returns the execution flow to it.
An interaction with DOS I/O and an addition of two 16-bit integers with carry were implemented in native code.
Frankly speaking I don’t find it feasible to implement a full 16-bit adder using Boolean functions only. The original NOR interpreter only performed bitwise NOR. It wasn’t possible to move/shift an individual bit to another one but the full binary adder needs it to consider the carry.
Recently I’ve come back to this NOR interpreter approach but in Python world. I’ve modified the original idea a bit to simplify (or just make it possible) an implementation of addition using NOR only, without native code at all. I’ve introduced one extra operation in the interpreter ñ the result of NOR is cyclically shifted left by 1 bit and stored the special location in memory.
def norcpu(): while 1: i = mem[IP]; a = mem[i + 0] b = mem[i + 1] r = mem[i + 2] mem[IP] = i + 3 f = nor(mem[a], mem[b]) mem[r] = f mem[S] = ((f >> 31) & 1) | ((f & 0x7FFF) << 1)
There are two special locations now: IP
(instruction pointer) and S
(shift register).
Let’s try to implement the full addition of 16-bit words with carry. I will use a simple macro assembler.
The full 1 bit adder formulas are:
sum = (a ^ b) ^ carry
carry = (a & b) | (carry & (a ^ b))
Now in the NOR CPU assembly:
; Input:
; mask - a current bit mask (0x0001, 0x0002, 0x0004, 0x0008 etc)
; carry ñ a carry from the previous bit (the masked applied)
; a, b - an argument addresses
; r - an address of the result
; Output:
; r - a result
; carry - a carry to the next bit (already left shifted with respect to the mask)
; mask - a mask left shifted by one bit
;
; Variables with '@' prefix are local for the macro.
;
!macro FADD mask, carry, a, b, r
XOR a, b, @t1 ; Formula: sum = (a ^ b) ^ carry.
XOR @t1, carry, @bit_r ;
AND @bit_r, mask, @bit_r ; Mask all bits in @bit_r expect the current one
OR @bit_r, r, r ; Save the bit to the result: r |= sum
AND a, b, @t2 ; Formula: carry = (a & b) | (carry & (a ^ b))
AND carry, @t1, @t1 ;
OR @t2, @t1, carry ; The carry is calculated. Its left shifted copy is in S.
MOVE S, carry ; Assign the carry to itself but shifted the next bit.
MOVE mask, mask, mask ; Dummy assignment to just get: S = mask << 1.
MOVE S, mask ; mask = S = mask << 1
Then we implement the 16-bit adder:
; Input:
; a, b - argument
; carry ñ a carry (the least significant bit only makes sense)
; Output:
; r - the result: r = a + c + carry
; carry ñ a carry (the least significant bit only makes sense)
;
; Variables having '@' prefix are local for the macro.
; const_1 - a special location containing the constant 0x0001.
;
!macro ADC a, b, carry, r
XOR r, r, r ; r = 0.
MOVE const_1, @mask ; The initial mask value = 0x0001
*16 FADD @mask, carry, a, b, r ; Repeat FADD 16 times (no loops, just a
; repetition)
AND carry, const_1, carry ; Clean-up all bits in carry except LSB.
What happens in ADC
? On each iteration (repetition) of FADD
it sums the current bit (its mask is @mask
). Then the sum is joined (via OR
) with the result. The mask is shifted to left 1 bit (the @mask
takes values 0x0001 -> 0x0002 -> 0x0004 etc). The carry is also shifted to the left 1 bit to be used on the next iteration of FADD
. After 16 iterations the carry will be in LSB (because the interpreter shifts cyclically). The carry after 16 iterations will be output value of carry.
The addition is implemented eventually. Then we could programmatically implement a subroutine stack. CALL
/RET
instructions could be based on the stack mechanism and branching instructions.
Now we can easily do any computations on our pseudo CPU executing only one command.
What’s the point of all this stuff? First of all is an academic interest, fun.
Related posts:
a += b;
b = a - b;
a -= b;
This is neat but unlikely useful in practice. But it could be rewritten using bitwise arithmetics to avoid overflow issues:
a ^= b ^= a ^= b;
This one looks very cool and hackish but unfortinatelly may cause an unpredictable behavior according to the Standard due to the chained assignments. So the last edition is also fast and overflow free but doesn’t have an unpredictable behavior:
a ^= b;
b ^= a;
a ^= b;
Update: Here is a perfect link to Bit Twiddling Hacks with lots of similar bitwise tips.
I love to use Sieve of Eratosthenes as “Hello, world!” when exploring a new language.
Here is my version of the seive in Go:
File erato-go-bool.go
:
package main import "fmt" import "math" import "flag" func main() { var N int flag.IntVar(&N, "N", 100, "") flag.Parse() fmt.Printf("%d\n", N) seive := make([]bool, N) limit := int(math.Sqrt(float64(N))) + 1 for i := 2; i < limit; i++ { if !seive[i] { for j := i * i; j < N; j += i { seive[j] = true } } } count := 0 for i := 2; i < N; i++ { if !seive[i] { count++ } } fmt.Printf("%d\n", count) }
But how fast is this?
I’ve compared it against implementations in C++ and C.
The first competitor is Go using bool
type in as a storage (see above). The second one is the also Go’s version but with int
as the storage type.
File erato-go-int.go
:
package main import "fmt" import "math" import "flag" func main() { var N int flag.IntVar(&N, "N", 100, "") flag.Parse() fmt.Printf("%d\n", N) seive := make([]int, N) limit := int(math.Sqrt(float64(N))) + 1 for i := 2; i < limit; i++ { if seive[i] == 0 { for j := i * i; j < N; j += i { seive[j] = 1 } } } count := 0 for i := 2; i < N; i++ { if seive[i] == 0 { count++ } } fmt.Printf("%d\n", count) }
Then I tested in C++. A TYPE
macro allows to compile the source with different types (int
and bool
):
File erato-cxx.cpp
:
#include <iostream> #include <vector> #include <cstdlib> #include <cmath> int main(int argc, char* argv[]) { int n = argc > 1 ? std::atoi(argv[1]) : 100; std::cout << n << std::endl; int sqrt_n = static_cast<int>(std::sqrt(static_cast<double>(n))) + 1; std::vector<TYPE> S(n, true); for (int i = 2; i < sqrt_n; ++i) if (S[i]) for (int j = i*i; j < n; j+=i) S[j] = false; int count = 0; for (int i = 2; i < n; ++i) if (S[i]) count++; std::cout << count << std::endl; return 0; }
And to have the full picture there is an implementation in C:
File: erator-c-int.c
:
#include <stdio.h> #include <stdlib.h> #include <memory.h> #include <math.h> int main(int argc, char* argv[]) { int n = argc > 1 ? atoi(argv[1]) : 100; int* S; int count; int sz = n * sizeof(*S); int i, j; printf("%d\n", n); long sqrt_n = sqrt(n) + 1; S = malloc(sz); memset(S, 0, sz); for (i = 2; i < sqrt_n; ++i) if (S[i] == 0) for (j = i*i; j < n; j+=i) S[j] = 1; count = 0; for (i = 2; i < n; ++i) if (S[i] == 0) count++; printf("%d\n", count); free(S); return 0; }
Makefile for easy run:
File Makefile
:
.SILENT: all: $(MAKE) run 2>&1 | tee log $(MAKE) parse-log run: go-bool go-int cxx-int cxx-bool c-int N ?= 100000000 go-bool: echo $@ 6g erato-$@.go 6l -o erato-$@ erato-$@.6 time -p -f %e ./erato-$@ -N=$(N) go-int: echo $@ 6g erato-$@.go 6l -o erato-$@ erato-$@.6 time -p -f %e ./erato-$@ -N=$(N) cxx-bool: echo $@ g++ -o erato-$@ \ -O3 -funroll-all-loops -fomit-frame-pointer \ -DTYPE=bool erato-cxx.cpp time -p -f %e ./erato-$@ $(N) cxx-int: echo $@ g++ -o erato-$@ \ -O3 -funroll-all-loops -fomit-frame-pointer \ -DTYPE=int erato-cxx.cpp time -p -f %e ./erato-$@ $(N) c-int: echo $@ gcc -o erato-$@ -lm \ -O3 -funroll-all-loops -fomit-frame-pointer erato-$@.c time -p -f %e ./erato-$@ $(N) parse-log: printf "%10s %10s %8s %5s\n" "Language" N Count Time ; \ (echo "------------------------------------") ; \ while read type ; do \ read N && \ read count && \ read time && \ printf "%10s %10s %8s %5s\n" $$type $$N $$count $$time ; \ done < log
I run this on Ubuntu 64-bit. The C/C++ compiler is gcc 4.4.1. The Go compiler is the lastest from its official repository.
Run:
make N=100000000
Output:
Language N Count Time
------------------------------------
go-bool 100000000 5761455 3.96
go-int 100000000 5761455 6.58
cxx-int 100000000 5761455 6.76
cxx-bool 100000000 5761455 2.20
c-int 100000000 5761455 6.47
C++ using std::vector<book>
has beaten C and Go. The second is Go’s implementation also using bool
. And on the third place are C, C++ with std::vector<int>
and Go with int
.
A straightforward solution could be:
#include <iostream> #include <vector> int main(int argc, char* argv[]) { int a[] = { 1, 2, 3, 4, 5 }; std::vector<int> v(a, a + 5); for (int i = 0; i < v.size(); ++i) { std::cout << v[i]; if (i < v.size() - 1) std::cout << ", "; } std::cout << std::endl; return 0; }
A condition in the loop solves the problem, it’s better to iterate containers using manipulators:
#include <iostream> #include <vector> int main(int argc, char* argv[]) { int a[] = { 1, 2, 3, 4, 5 }; std::vector<int> v(a, a + 5); for (std::vector<int>::const_iterator i = v.begin(); i != v.end(); ++i) { std::cout << *i; if (v.end() - i > 1) std::cout << ", "; } std::cout << std::endl; return 0; }
It’s not also great because iterators of some containers don’t support subtraction. For example, this code doesn’t compile if we use std::list
instead of std::vector
(the first example also fails on std::list
but due to another reason). The better way is:
#include <iostream> #include <vector> int main(int argc, char* argv[]) { int a[] = { 1, 2, 3, 4, 5 }; std::vector<int> v(a, a + 5); typedef std::vector<int>::const_iterator iterator; for (iterator i = v.begin(); i != v.end(); ++i) { std::cout << *i; if (std::distance<iterator>(i, v.end()) > 1) std::cout << ", "; } std::cout << std::endl; return 0; }
The std::distance
template can calculate the distance between two iterators even if they don’t support arithmetics. But for this kind of containers std::distance
just iterates from one to another and the overall time of the print loop seems to be O(N^2)
instead of original O(N)
. Also we have to use the type name twice — to declare the iterator and to instantiate std::distance
. For example, Visual Studio 2008 cannot deduct the type from the parameters of std::distance
.
There is another neat way allowing to use iterators (with O(N)
time even for containers such as std::list
), and to write beautifully and compactly:
#include <iostream> #include <vector> int main(int argc, char* argv[]) { int a[] = { 1, 2, 3, 4, 5 }; std::vector<int> v(a, a + 5); for (std::vector<int>::const_iterator i = v.begin(); i != v.end(); ++i) { std::cout << *i; if (i != --v.end()) std::cout << ", "; } std::cout << std::endl; return 0; }
The trick with pre-increment --
operator allows to check effectively for the last element of a container.
sizeof
for composite types (structures and classes) before digging in to the alignment is to remember that sizeof returns the difference between addresses of two consequent elements of an array containing instances of the type.
I will put the latest features and hot bug fixes there, and also merge back the cmockery trunk as well.
Related posts:
Stephen Dewhurst, “C++ Common Knowledge”
and in general, this training is specifically focused on the topics from it.
Very cool man. It was interesting and accompanied with jokes like that Boost guys just smoke templates etc. I really liked it.
Steve said that C++ is virtually the most that is doing in life. He wrote the compilers, utilities, versed in the standards and now he provides trainings.
I cannot say that I learned something very new - it would be strange since I read his book above from cover to cover and periodically get back to it. Although perhaps one thought I was hooked: the correct implementation of the copy constructor or assignment operator for the class in the hierarchy with a virtual base class having data members is a very complicated task. This clearly violates the principle of independence of the logical levels in the hierarchy of inheritance, as it necessary requires to know exactly from which classes you inherited and how to properly initialize them with multiple inheritance.
The recommendation is as follows: first ask yourself a question: do I need a multiple inheritance here? do I need a virtual multiple inheritance?? but do I need a virtual multiple inheritance with the data in the virtual base class??? And even after long reflection it’s better to say “no”. Personally I don’t have anything against the multiple inheritance. But I don’t really like how it’s implemented in C++. And I don’t really like how it’s done in Java either. I do like how it’s done in Go. In Go the notions of data structures and interfaces are completely separated. Data structures cannot be inherited. They can only implement interfaces. And you can inherit only an interface. Therefore, in principle, it isn’t possible to pick up other people’s data in inheritance but only methods. And no data, no problem in its initialization.
So I just summarize the general recommendations from Steve:
Peter Seibel, “Coders at Work”
The book consists of interviews with a dozen well-known programmers. Here are creators of UNIX, Netscape, JavaScript, Smalltalk, Haskell, Erlang, Ghostscript, LJ and also merely Donald Knuth.
The author asks them similar questions: when and why you started programming, how you usually work, where and on what you have worked and are working now, what do you think about the development of programming languages over the past decade, what you can advise the young etc. Some memoirs theme is dangerous because you can slip into elementary senile grumbling like “but at our times…” or “you’d better write in machine language and it’ll teach you more…”, but all proved to have very balanced view of reality. Of course, there is a radical separation between functional and imperative fans but this is the question of religion rather than age.
Many refer to various books - I have expanded much my “must read” list.
It’s funny that almost no one answered unequivocally positive about C++. They were like “very hard, difficult etc” or “well, now that more than anything so far there is no better than to build native code of industrial complexity, let him be.”
Lyrical digression. I’ve been smoking Go and it takes me off deeper and deeper. I can say that I have almost found all of my C++ habits in Go. And its innate multi threading and ultra-fast compile polish everything.
Also there is an interesting opinion on whether it’s required for all self-respecting programmers to read the “Art of Computer Programming” Knuth, or at least have in the library. Many recognized that they didn’t read from cover to cover, but used as a reference.
As always I’ve got that I did not even have heard of some very famous things. For example, Literate programming from Donald Knuth or Bloom filter.
In general, I’ve found the book very interesting. If you are longing covers like “should I program the entire career…” or “should I shift to management or architects because young is pressing from the back…”. Here is given an excellent, but the hidden answer: any of these ways can bring and satisfaction and, importantly, wealth. That’s the beauty of our profession. Just do the thing that you want to work till night, look around wondering what’s going on and not worry that you can be unnecessary - you can’t.
As a consequence of the fact that every commit is prepared, debugged and tested a sizable time (the benefit is that there are bug fixes only which are typically small) and even formal aspect of an issue may take a couple of days, often happens that when it comes directly to the command “commit” all ends up with a conflict because someone has already managed to touch a piece of your code and pour it on the server. Then it needs to merge manually. And if the file is not one then a headache begins.
Since I am only recently involved in this stuff after the second commit I’ve decided to simplify my life in terms of merging on a conflict.
I’ve got the git and now it looks this way: each bug fix lives in the separate git’s repository (in fact, in a directory) with two main branches. In one I do patching and preserve entire history of it in git, and periodically I sync the second branch with the main repository. And if I’ve got new just synced changes in the second branch I merge the first one with it using just one magic command git merge
.
In terms of distributed SCMs I now mostly with the mercurial because Google Code supports it. But frankly speaking the git is extremely powerful tool (of course if Windows is not involved in the workflow because git’s Windows port is horrible).
At the beginning of using git there is lots of confusion. Personally I was absolutely misleading by the idea of the staging area (or the index). This is an intermediate chain between local files and the repository. So git diff can show the three different things: the diff between the local files and the index (but not the repository as many people expect by default and it introduces git’s newcomers to a screeching halt), the diff between the index and the repository and finally the diff between the local files and the repository. The index (or the staging area) allows to commit changed files selectively. Usually a commit affects only staged files (in the index). And the most interesting it is possible to stage (and eventually to commit) files partially. For example, I’ve added two brand new classes to a file but I am able to commit only one and commit the next one later.
Do you already like it?
For example, a rollback of all local changes could be done at least two ways (using git checkout or via git reset). Also to rollback of already committed change you also have as minimum two alternatives (git reset или git revert) depending on your wish to make this rollback visible in the history.
The abundance of features and some of their dissimilarity to the generally accepted standards of SCM commands is a little daunting at the beginning. But after a while you get started to feel the git’s power. For example, having the staging area and git stash (when possible to freeze the state of local changes, do some quick hacking and then get that state back) are very unique features of git.
In terms of GUI the gitk provides all the necessary.
The only thing that you need to choose on taste alone is program for merging in a graphical mode to resolve conflicts. Here all are in their preferences.
I would recommend a very good book to start with git.
Travis Swicegood, “Pragmatic Version Control using Git”
This is the very good introduction into Git from the community of Linux kernel developers and from Linux Torvalds in particular.
I appreciate these books for primary involvement in the subject.
This is a book for beginners, and if you’re not a novice in the field of DVCS, then you swallow it in one evening and want a more in-depth knowledge of Git. It happened to me. I read the book for the evening, it formulated dozens of unanswered questions and allowed me to understand - what kind of questions I need answers for.
You will find the excellent live book about Git on the official website. Many chapters are accompanied with video tutorials.
I usually do not keep the book for beginners because after you are interested in the subject any more or dig deeper and knowledge for beginners becomes useless. But I leave this one.
Using right and convenient tools greatly speeds work. And the time spent at the beginning to choice and to set up them definitely pays off in the future.
Hope I was able to attract into the ranks of git users a few more enthusiasts.
Once accustomed to the constant presence on hand a version control, you want it to be everywhere. Even when repairing a car.
I could send it off to all interested in the order of requests. Naturally it is relevant only if you have a Gmail account. So please please indicate an address at Gmail to send the invitation.
Please be patient because after the invitation is sent it could take a couple of days until you get it.
And one more thing – share your own invitations because Wave is cool where there are many people in it.
Just as in the good old days but now it works in a browser.
The latest version is 0.6. Apart from the emulation there are the built-in assembler to write and compiler a code for Intel 8080 directly in the window of the emulator, and the almost interactive disassembler to view not only a code but also data.
A few screenshots (click-able):
Emulator (“Volcano” game):
Assembler:
Disassembler:
The list of games is also being updated.
I do support only Chrome really but they say it also works in Firefox and Safari with more or less of little glitches.
It is difficult to explain my fun from this project. This is something deep inside.
class Date { public: Date(int year, int month, int day) { ... } };
Unfortunately not everybody in the world uses the same quite logical notation of Year/Month/Day or Day/Month/Year. Some people prefer Month/Day/Year. But even the first two could be easily mixed up. If the following is written:
Data d(2009, 4, 5);
Is it 4th of May or 5th of April? Who can be entirely sure what it exactly means without looking in the class declaration?
Any chance to improve the design? Indeed.
For instance:
class Year { public: explicit Year(int year) : year_(year) {} operator int() const { return year_; } private: int year_; };
And similar for the rest:
class Month { ... }; class Day { ... };
Now the interface could look this way:
class Date { public: Date(Year year, Month month, Day day); Date(Month month, Day day, Year year); Date(Day day, Month month, Year year); }
We can instantiate the class as:
Date d(Year(2010), Month(4), Day(5));
or
Date d(Month(4), Day(5), Year(2010));
The result is always visible and fully predictable in a calling code. Everything will be inlined so no slow down involved at all because of those three “unnecessary” classes.
I agree there is more typing job but it fully gets you rid of any typos and consequently of silly but annoying bugs.
Any objections?
I was debugging a brand new online assembler for my Radio-86RK emulator. That debugging meant some dancing around HTML.
To build a final HTML file from a bunch of tiny files I used a very simple program. Here is a bit of code from it:
... while (!feof(f)) { char line[1024]; *line = 0; fgets(line, sizeof(line), f); printf(line); } ...
Implied that this code should copy all lines from a file f
to the standard output.
Even if we don’t care about using a buffer with a constant length and rest of other C-like features, this code has one serious drawback which embarrassed me for a quite awhile. It worked okay until I had started to play with percent widths and heights of HTML objects.
Instead of getting:
<table width="100%">
I was ending up with:
<table width="100">
You have probably already guessed why. But to tell the truth I had been investigating this up to half an hour.
So instead of:
printf(line);
I had to write:
printf("%s", line);
Otherwise all percentage characters are treated as formatters because the first parameter of printf()
is a format and all non-escaped %
characters will be deleted. That is what was happening to me.
Conclusion (following after the first one - “serves you right”): It is much safer to write in C++ and use STL streams for formatting.
I have come over today and see this:
There is a new item in the Source tab — Close (in red). This is what we have been waiting for a long time — project cloning.
If an owner of some project does not want to make you as a contributor but you are desperately keen to show off your work on the project, you can simply clone it and keep going to work on your own fork. Entire history of an original project will be fully inherited.
A clone can be cloned as well.
They cover not only the framework itself and samples of using it but also there is a couple of words about the test driven development methodology. There are some basic recommendations given for writing testable code.
In whole it would be not worse ten minutes to have a look and give it some thoughts.
This bit is roughly one year old project — the remote control over Bluetooth for Lego NXT using the applet running on a mobile phone.
I’ve called it unpretentiously — nxtbtrc.
Everything is simple. The Java applet running on a J2ME compatible phone pairs with Lego NXT brick and then sends commands to it. Nothing special but it was interesting to figure out how to use Bluetooth API in J2ME.
I am not sure about any further development but who knows — maybe it comes in handy for somebody.
Here is the small video demonstrating its work:
I have even bought the book for this case. By the way the book is quite nice. It tells about Bluetooth from the programmer’s point of view in a simple and clean manner. A few stacks of different vendors are covered, their comparison is given and accompanied with examples on different languages and platforms.
Albert Huang, Larry Rudolph, “Bluetooth Essentials for Programmers”
Any changes? Indeed.
The major feature coming is the “The event listener API”. In other words this is the way to customize the output produced by tests without any code changes in the library sources.
For example, the standard output of the trivial test (file runner.cpp
):
#include "gtest/gtest.h" TEST(One, Simple) { EXPECT_EQ(1, 2); } int main(int argc, char** argv) { ::testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); }
is
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from One
[ RUN ] One.Simple
runner.cpp(4): error: Value of: 2
Expected: 1
[ FAILED ] One.Simple (15 ms)
[----------] 1 test from One (15 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (31 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] One.Simple
1 FAILED TEST
To customize the output we have to implement our own event listener (for instance, file runner.cpp)
:
#include "gtest/gtest.h" using namespace ::testing; // Our own event listener. class LaconicPrinter : public ::testing::EmptyTestEventListener { // Called before a test starts. virtual void OnTestStart(const TestInfo& test_info) { printf("*** Test %s.%s starting.\n", test_info.test_case_name(), test_info.name()); } // Called when an assert fails or SUCCESS gets called. virtual void OnTestPartResult(const TestPartResult& test_part_result) { printf("%s in %s:%d\n%s\n", test_part_result.failed() ? "*** Failure" : "Success", test_part_result.file_name(), test_part_result.line_number(), test_part_result.summary()); } // Called after a test finishes. virtual void OnTestEnd(const TestInfo& test_info) { printf("*** Test %s.%s ending.\n", test_info.test_case_name(), test_info.name()); } }; TEST(One, Simple) { EXPECT_EQ(1, 2); } int main(int argc, char** argv) { ::testing::InitGoogleTest(&argc, argv); // Obtain the reference to the active listeners. ::testing::TestEventListeners& listeners = ::testing::UnitTest::GetInstance()->listeners(); // Delete the default event listener. delete listeners.Release(listeners.default_result_printer()); // Add our LaconicPrinter listener. Google Test will take care about it. listeners.Append(new LaconicPrinter); return RUN_ALL_TESTS(); }
Now the output looks as:
*** Test One.Simple starting.
*** Failure in runner.cpp:31
Value of: 2
Expected: 1
*** Test One.Simple ending.
Note that it could be several active listeners. But their output can be messed. Therefore we firstly remove the default listener from the list to avoid in influence on our listener’s output.
In general the event listener interface has these methods:
class EmptyTestEventListener : public TestEventListener { public: virtual void OnTestProgramStart(const UnitTest& unit_test); virtual void OnTestIterationStart(const UnitTest& unit_test, int iteration; virtual void OnEnvironmentsSetUpStart(const UnitTest& unit_test); virtual void OnEnvironmentsSetUpEnd(const UnitTest& unit_test); virtual void OnTestCaseStart(const TestCase& test_case); virtual void OnTestStart(const TestInfo& test_info); virtual void OnTestPartResult(const TestPartResult& test_part_result); virtual void OnTestEnd(const TestInfo& test_info); virtual void OnTestCaseEnd(const TestCase& test_case); virtual void OnEnvironmentsTearDownStart(const UnitTest& unit_test); virtual void OnEnvironmentsTearDownEnd(const UnitTest& unit_test); virtual void OnTestIterationEnd(const UnitTest& unit_test, int iteration); virtual void OnTestProgramEnd(const UnitTest& unit_test); };
Also there is the brand new flag --gtest_shuffle
allowing to run tests in a random order. Via --gtest_random_seed=SEED
flag it becomes feasible to control the randomness of this order. If SEED is 0 the current time is used to initialize the random generator.
Reporting is going to be more compatible the JUnit. Using --gtest_output
allows to generate reports which can be easily picked up JUnit compatible tools, for instance, Hudson.
Very cool that now in Visual Studio each failing test reports are duplicated to the standard “Output” window. It is very nice that this change is based on my code.
In addition the --gtest_print_time
option is turned on by default. Very handy.
There are also some minor improvements:
tuple
implementation getting rid of the boost dependency when Compile()
is usedThe joy, no doubt!
I have stopped to develop anything without tests long time ago and Google Test makes this process easier and faster.
I have already upgraded to the version 1.4.0, and you?
The first computer I started programming on was Radio-86RK. My brother built it… and it got started and keeps going up to nowadays.
That is why I still have a tender affection for this little 8-bit piece of hardware. As the result of this affection I have been writing its emulators.
The first one was for DOS. I still keep its original web site without any changes. That emulator was quite powerful: the built-in debugger, the game cracking mechanism etc. But DOS time was over and now that emulator works properly only in DosBox. The sources are available for download.
The next generation of dear RK was for Windows and based on SDL. But there was no the built-in debugger and the project itself seems to be unfinished but runnable and allowing to still play games. That is why its binary only but accompanied with bunch of games is available on public.
But a couple of days ago I have come across this - the emulator of Sinclair ZX Spectrum written on pure JavaScript (no applets, activex etc.)
I was impressed and inspired so after a day or two my old little RK monster has been born again and its new platform is JavaScript. It seems that proper browsers now already provide quite good JavaScript performance. 2D graphics are implemented via canvas
HTML5 tag.
The project comes out as Radio-86РК in JavaScript (in Russian).
The emulator and games live in the one single file radio86.html. By clicking on this link the emulator gets started right in a browser. There is the game selector at the bottom, and the possibility to play around with screen dimensions and speed. Even if you do not understand Russian there is no problem at all. Just launch the emulator, select a game and try to play. Most of games use the arrows for movements and the space bar to shoot/jump/take/etc. You will definitely feel the spirit of those 8-bit B&W dodgy games. Have a fun!
In general the emulation works on the Intel 8080 commands level.
Here is the screenshot of the classic game Volcano from the emulator.
At the moment I’ve tested the emulator in the Google Chrome 4.* only. I’m not quite keen about any compatibility with other browsers but let’s see how it goes along. IE (even version 8) does not definitely handle it properly but Firefox and Opera could try.
The wonderful 8-bit world of Radio-86RK is coming back!
Update: Version 0.3 is released. It works much faster and does not thrash CPU anymore. Also I have included a few applications (interpreters, compilers, tools etc.)
const T*
to declare a pointer is exactly the same to T const*
because it’s only important here to use const
before *
but the order of T
and const
doesn’t matter.
The both:
const T* p;
and
T const* p;
declare the pointer p
to a const object but not a const pointer. The pointer itself can be modified:
T const* p; ... p = NULL;
But it’s impossible to modify the object:
T const* p; ... p->some_member = 0; // ERROR: error C2166: l-value specifies const object
This was the introductory and let’s talk about the topic.
I’m really keen to have readable sources. I could be wrong but in my opinion from the universal point of view using const at the beginning of an expression (for, instance, const T* p;
) implies that the entire expression is const
. It doesn’t really matter that in this particular case according to the C++ rules it means only that the object is const
but not the pointer.
That is why T const* p;
could be treated differently, notably “a type T
which is const
and the pointer to this type”. Readability becomes a bit better.
Of course, all these are questions of a coding style and it’s always recommended to be consistent modifying existing sources. But if you’re just learning or beginning a new project or something when it’s feasible to try something fresh, it may worth not to refuse yourself in this experience.
For instance, the specifics of the development in our company are that we support and develop our products on many platforms such as Linux, SunOS, AIX, HP-UX, Tru64, Windows etc. Talking about platforms, even for UNIX family the development of portable software is quite a challenging problem. For example, multithreading and networking are not straightforward enough to be implemented portably right away. Every time a new piece of code or a patch is coming up it must be verified on all supported platforms starting from just compilation up to unit testing and QA.
Unfortunately Perforce is not quite flexible in this instance. Here is an example of the obvious development cycle: The piece of code is initially developed and tested on one machine, therefore any added, changed, deleted or integrated files are in the “submit pending” list on this machine. When the developer is about to check it in, unfortunately for him and fortunately for other team members, he must check his new code on other development boxes. Perforce does not provide the mechanism of the “soft-submit” or allow temporary check-in of the new code with subsequent check-out on another machine for testing purposes. This process also becomes really painful when the amount of the pending code is quite large, the number of machines where you have to check the code is also not just one or two, and on top of it the number of iterations for “change-submit-check-revert” is exceeding a few dozens per day.
The tool I am going to tell you about aims to salvage this issue. It allows transferring your current Perforce pending list from one development box to another in split second.
The following assumes that the reader is a bit familiar with using Perforce.
Of course a normal UNIX user will offer some kind of simple scripts to automate the process, which was my thinking initially as well. Unfortunately the scripting is not 100% portable due to so many different shells involved (ksh, bash, csh). If that wasn’t enough unfortunately Windows “scripting” is also involved into the process, and its behaviour is far from UNIX way even with the use of Cygwin. My first version of the tool was written on Korn shell but it was not really portable between UNIX and Windows. The second version was written on Python and it would be just fine apart from Python not being easily available on all development boxes in case the development involves a little bit dodgy platforms such as HP-UX and AIX.
Finally I ended up with the third version completely rewritten on pure ANSI C. The final name is p4p
(P4 Patcher). The idea is for the tool to be distributed in source and compiled on the target platform just before use. Also, the tool is targeted to provide maximum debugging and error related information on the runtime. It helps the end user (the developer) to fix the tool if it does not work properly on any particular platform. I have been successfully using the tool on Linux, SunOS, AIX, HP-UX and Windows. The most exciting thing is that I can easily transfer the pending change list between all these platforms.
I put the source to the Google Code - p4patch and it can be checked out from there. You will then have to compile and run it on other development machines.
So let’s step away from the theory and look into something real. Let’s assume you have a bunch of files ready to submit on your box. The Perforce p4 opened
command shows that list.
Here is the crush course for your next steps:
Linux, AIX, HP-UX:
./compile-unix.sh
Solaris:
./compile-solaris.sh
Windows:
compile-vs2008.cmd
Run the p4patch server on the remote machine (e.q. 192.168.1.9):
p4p server
Generate a patch archive on the local machine via p4p diff
(patch.tar
will be created). This command uses p4 opened
to get the list of the files included into the patch and packs them into a TAR archive. The file list can be provided explicitly via -o
option.
By p4 opened | grep ...
you can generate your own customized list.
Apply the patch on the remote machine:
p4p patch -h 192.168.1.9
List opened files on the remote machine:
p4p exec -h 192.168.1.9 -p4 opened
See the diff on the remote machine:
p4p exec -h 192.168.1.9 -p4 diff
Revert a patch on the remote machine:
p4p revert -h 192.168.1.9
Display the P4 version on the remote machine:
p4p exec -h 192.168.1.9 -p4 -V
While p4p server is running on the remote machine you can fully control the remote P4 client and also apply/revert your patch. The patch itself is the standard TAR file containing files affected by your patch and the file list.
When p4p applies the file change on the remote machine it always tries to preserve the current line ending used on the remote platform.
If several people work on one machine in parallel they cannot share the same TCP/IP port for the p4p listener. They need their own port number. The default p4p port number is 20050. If you want to use your own port, follow these commands (i.e., assuming that you plan to 30001 as your port):
Run the server on your port:
p4p server -l 30001
Client side:
Apply the patch on the remote machine:
p4p patch -h 192.168.1.9 -p 30001
List opened files on the remote machine:
p4p exec -h 192.168.1.9 -p 30001 -p4 opened
See the diff on the remote machine:
p4p exec -h 192.168.1.9 -p 30001 -p4 diff
Revert a patch on the remote machine:
p4p revert -h 192.168.1.9 -p 30001
Display the P4 version on the remote machine:
p4p exec -h 192.168.1.9 -p 30001 -p4 -V
If your find that p4p cannot start the server perhaps somebody else uses the same port. Choose another port is this case. p4p started with -?
switch prints all options available.
Finally I would give my point of view on the workflow using Perfoce on several boxes at the same time.
I always try to preserve the cross-platform consistency of a change set. It means when I have finished to polish a change set on my main working box (let’s say Windows) and it works okay, then I use p4p to push this change set to another box (say Linux) and debug it there. If any changes needed for Linux I do it on the MAIN box (here it is Windows) and then push it again to the Linux. p4p works almost instantly so this process of pushing changes from the main box is very comfortable. Then I step to the next platform and so on (HP-UX, AIX etc). When I get it working on all boxes (it works, tests are passing etc.) I submit that change set from the MAIN box.
I try to avoid working on the same change set on different platforms (I mean “working” as doing source changes).
Hope all the this will help to improve your Perforce workflow.
Though my programming experience is more than ten years and it covers various languages from machine codes and assembler up to the functional programming I have discovered the test driven development world quite recently. Programmers are often very conservative (and quite lazy!) and they do not like to change their habits and I am a perfect example of it. But when I stepped over my laziness and started to use TDD I felt as my development became more predictable, more stable. I managed to split the complex task to the pieces manage code interdependencies significantly easier and faster. More importantly: I have stopped repeating my coding mistakes, reintroducing already fixed bugs and now I am able to refactor my code anytime without any fear of breaking something important a day before the release. Why? All thanks to the test driven development.
I would like to share my experience on entering to the wonderful world of TDD and hope to encourage somebody to join.
My main background is C and C++ that’s why I will cover these languages but all ideas mentioned are common for lots of modern languages (Java, C#, Python, Delphi etc).
Let’s start from the beginning. Usually the first program written by a newbie is Hello World. Assume you have done it already and you want to do something more complex.
Let’s assume you studied a lot of computer science and you know how to implement very fast multiplication function and you have done it. This is what it would look like:
File: mult.h
:
#ifndef _MULT_H #define _MULT_H int mult(int a, int b); #endif
File: mult.cc
:
#include "mult.h" int mult(int a, int b) { if (!a || !b) return 0; int r = 0; if (a == 920 && b == 847) r++; do { if (b & 1) r += a; a <<= 1; } while (b >>= 1); return r; }
I want to warn the reader that this particular example is not ideal in terms of the coding style and it’s not clear in logic, it uses a lot of C/C++ “cool short” expressions and so on. Also the function has some weird line with 920 and 847. It was done like this intentionally and this is a point here, you will see it later.
Now, you have done the code. You definitely know that it should work more reliably and faster because your computer science background tells you that. How can you make sure that it works fine? The function code is quite “non-understandable” and you cannot swear that it works correctly just by looking on the source. You have to try it on. The first and the most obvious way to create the simple example like this:
#include "mult.h" #include <iostream> int main(int argc, char* argv[]) { while(1) { std::cout << "enter a: "; int a; std::cin >> a; std::cout << "enter b: "; int b; std::cin >> b; std::cout << "a * b = " << mult(a, b) << std::endl; } }
Then you run it, play it a bit, try a couple of examples and then make the conclusion that it works. Later you add the mult.cc
file to your project and probably delete the test example source because you do not need it anymore. You have linked the function into your application and you are almost happy.
Let’s step back for a second now and imagine that unfortunately sometimes your application shows wrong result or perhaps crashes and you suspect that the issue is your mult()
function. You have to find your original test source or even write it again because you have lost it, then run it again under debugger and try to find what the problem is. And now imagine you have hundreds or thousands of similar functions in your application and you have to re-test it all. It’s nightmare.
Well, let me show you another way – the test driven development way. We will use the excellent Google Test Framework 1.3.0 for that. You can download and unpack it in your working directory:
wget http://googletest.googlecode.com/files/gtest-1.3.0.tar.gz
gzip -dc gtest-1.3.0.tar.gz | tar xvf -
It will create gtest-1.3.0
directory in your current folder. We will refer to this directory below so make sure that you use proper directory names in your compilation commands.
Then you create the unit test.
File: mult_unittest.cc
#include <gtest/gtest.h> #include "mult.h" TEST(multTest, simple) { EXPECT_EQ(91, mult(7, 13)); }
This file contains the simple test case. The meaning of it is explained below.
Then the test main runner module:
File: runner.cc
#include <gtest/gtest.h> int main(int argc, char **argv) { testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); }
This runner will execute all declared tests in your test application. This piece of code can be almost the same for any of your unit test suites. It just parses the command line arguments and runs all tests.
Now let’s compile it. If you are running Linux and have the GCC C++ compiler version 3 or later you can use the following command:
g++ -Igtest-1.3.0/include -Igtest-1.3.0 -o mult_unittest gtest-1.3.0/src/gtest-all.cc mult.cc mult_unittest.cc runner.cc
The mult_unittest
executable should be generated. Let’s run it:
./mult_unittest
It prints something like this:
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from multTest
[ RUN ] multTest.simple
[ OK ] multTest.simple
[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran.
[ PASSED ] 1 test.
Let’s go back and look at it in more details now. We have created the test case named multTest.simple
in the file mult_unittest.cc
(multTest
is the test suite name and the simple is the test name in the suite) which runs your function with 7 and 13 as the parameters and checks that result is 91. The macro for the test declaration is TEST(...)
. The magic happens in the EXPECT_EQ (...)
. This function call has two arguments: the first one is the expected value and the second is the real one. If they are equal the function passes through quietly but if they are different it reports an error message.
The Google Test Framework provides a bunch of similar functions to check various conditions with different argument types. The EXPECT_*
function family does not abort the test run. It just prints the report about a test failure and keeps going to execute other tests. The ASSERT_*
functions (for example, ASSERT_EQ()
) stops the test suite run and terminates the runner. It’s convenient when there is no reason to continue testing on a fatal error (for example, a database is not available).
But in our case the test runner reports the successful test execution – the test case has been executed and the result is correct. That’s fine but this test case is so obvious and checks only one pair of numbers. You need more. Because the mult()
function has some weird checking of the argument for zero at the beginning let’s test it. You add one more test case – multTest.zero
.
File: mult_unittest.cc
#include <gtest/gtest.h> #include "mult.h" TEST(multTest, simple) { EXPECT_EQ(91, mult(7, 13)); } TEST(multTest, zero) { EXPECT_EQ(0, mult(0, 7)); EXPECT_EQ(0, mult(7, 0)); }
Let’s compile with the same command and run mult_unittest
executable again. It should print this:
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from multTest
[ RUN ] multTest.simple
[ OK ] multTest.simple
[ RUN ] multTest.zero
[ OK ] multTest.zero
[----------] Global test environment tear-down
[==========] 2 tests from 1 test case ran.
[ PASSED ] 2 tests.
The new test passes successfully as well and the mult()
function seems to handle checking parameter for zero correctly. But we still have unsolved issue – your application using the function mult()
fails and it means this function sometime returns wrong value. Let’s add the stronger test to file mult_unittest.cc
:
File: mult_unittest.cc
#include <gtest/gtest.h> #include "mult.h" TEST(multTest, simple) { EXPECT_EQ(91, mult(7, 13)); } TEST(multTest, zero) { EXPECT_EQ(0, mult(0, 7)); EXPECT_EQ(0, mult(7, 0)); } TEST(multTest, all) { for (int a = 0; a < 1000; ++a) for (int b = 0; b < 1000; ++b) EXPECT_EQ(a * b, mult(a, b)); }
This test (multTest.all
) checks all possible values of arguments from 0 to 999. Let’s compile and run it again:
[==========] Running 3 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 3 tests from multTest
[ RUN ] multTest.simple
[ OK ] multTest.simple
[ RUN ] multTest.zero
[ OK ] multTest.zero
[ RUN ] multTest.all
mult_unittest.cc:18: Failure
Value of: mult(a, b)
Actual: 779241
Expected: a * b
Which is: 779240
[ FAILED ] multTest.all
[----------] Global test environment tear-down
[==========] 3 tests from 1 test case ran.
[ PASSED ] 2 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] multTest.all
1 FAILED TEST
Wow! The test fails. It means we have found the problem. We see that in the line 18 of mult_unittest.cc
there is the test failure: the expected value is 779240 but the actual one is 779241. It’s a great result, but we also need to know which exact parameters cause this error. So let’s modify the test:
TEST(multTest, all) { for (int a = 0; a < 1000; ++a) for (int b = 0; b < 1000; ++b) EXPECT_EQ(a * b, mult(a, b)) << "wrong result on a=" << a << " and b=" << b; }
This code will also print the error message and the values of a and b on failure. The EXPECT_EQ(...)
can be used the output stream similar to std::cout
, for example, to print out the diagnostics on an test failure.
Compile and run it again. We should get the following result:
[==========] Running 3 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 3 tests from multTest
[ RUN ] multTest.simple
[ OK ] multTest.simple
[ RUN ] multTest.zero
[ OK ] multTest.zero
[ RUN ] multTest.all
mult_unittest.cc:17: Failure
Value of: mult(a, b)
Actual: 779241
Expected: a * b
Which is: 779240
wrong result on a=920 and b=847
[ FAILED ] multTest.all
[----------] Global test environment tear-down
[==========] 3 tests from 1 test case ran.
[ PASSED ] 2 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] multTest.all
1 FAILED TEST
Now we know exactly that the function fails when a=920
and b=847
. This is the problem. And now we can fix the “problem” by removing the line if (a == 920 && b == 847) r++;
from the mult.cc
file. Here is an error free version of the main.cc
:
#include "mult.h" int mult(int a, int b) { if (!a || !b) return 0; int r = 0; do { if (b & 1) r += a; a <<= 1; } while (b >>= 1); return r; }
Well, now compile it and run mult_unittest
once again. Here is the output:
[==========] Running 3 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 3 tests from multTest
[ RUN ] multTest.simple
[ OK ] multTest.simple
[ RUN ] multTest.zero
[ OK ] multTest.zero
[ RUN ] multTest.all
[ OK ] multTest.all
[----------] Global test environment tear-down
[==========] 3 tests from 1 test case ran.
[ PASSED ] 3 tests.
All tests work perfectly and now you are sure that your function mult()
is fully error free.
Let’s analyse what we’ve done. We have done the function mult()
and also we have created the tests which can be used any time to prove its proper functioning. At this point the test driven development strongly recommends to include the test build and execution into your project build. For example, this is the part of your myapp
project makefile:
... all: build build: cc –o myapp main.cc mult.cc
You should add the test compilation and run into this makefile:
... release: build test build: g++ –o myapp main.cc mult.cc test: g++ -Igtest-1.3.0/include -Igtest-1.3.0 -o mult_unittest \ gtest-1.3.0/src/gtest-all.cc mult.cc mult_unittest.cc runner.cc ./mult_unittest
Why do you need this? You need this because each time you release the project (using release
target) it will compile and run the test suite to make sure that the current implementation of the mult()
function is ok and works as we expect.
Now imagine you want to check whether it is reasonable to use your own hacky implementation of the simple arithmetic operation as the multiplication. Let’s run your test suite again by the command:
./mult_unittest --gtest_print_time --gtest_filter=multTest.all
We ask Google Test framework to print the test execution time and also we ask to run only one test using the filter by name.
The output:
Note: Google Test filter = multTest.all
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from multTest
[ RUN ] multTest.all
[ OK ] multTest.all (1266 ms)
[----------] 1 test from multTest (1297 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (1328 ms total)
[ PASSED ] 1 test.
It reports only one test run (testMult.all
) and it takes 1279 ms on my Core 2 Duo laptop (timing on your machine can be different).
Now you want to try another fairly simple implementation for the mult()
function:
File: mult.cc
#include "mult.h" int mult(int a, int b) { return a * b; }
Let’s compile it by exactly the same command as we used for the first implementation:
g++ -Igtest-1.3.0/include -Igtest-1.3.0 -o mult_unittest \
gtest-1.3.0/src/gtest-all.cc mult.cc mult_unittest.cc runner.cc
and run it:
./mult_unittest --gtest_print_time --gtest_filter=multTest.all
The output should look like this:
Note: Google Test filter = multTest.all
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from multTest
[ RUN ] multTest.all
[ OK ] multTest.all (1094 ms)
[----------] 1 test from multTest (1141 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (1171 ms total)
[ PASSED ] 1 test.
We see it takes only 1094ms on my laptop and it’s faster than our original handmade implementation.
Now you know that the original implementation is not quite so good and may be optimized or replaced by a better one.
So what is that we have achieved by this entire exercise? What is the message of it?
Firstly, we have created the test mechanism for our function and this mechanism can be used any time later to prove the function logic and it can be fully automated. Once created it can be re-used as many times as you want. You do not lose your efforts applied initially for creating the testing routine.
Secondly, we have included the test run into the project build. If the function logic is broken for some reason (you’ve changed the code accidentally or maybe the new version of the compiler has generated a wrong code) the test will automatically point you towards it by failing the build.
And thirdly, we tried two different implementations of the mult()
function using the same test suite. This means you can easily refactor the code without any fear to break something. The tests will check the function results and the expectations from the function. You have determined the function behaviour via the test cases and from this point you can easily play with the function implementation. On top of this we have tested two different implementations for execution time and now we have enough information to choose the better one.
These are really awesome results – you have automated the error checking procedure for your project. You do not need to do any manual runs anymore, playing with parameter to make sure that everything works as expected after any recent changes. Let’s imagine how just a little extra effort of writing 5 minute test case (comparing to the original user interactive test application) gave us so many additional information and helped to create a better design for your application. It’s definitely worth it.
There is probably an argument that in some cases testing can be tricky because the real world applications are much more complex than this isolated example. That is 100% correct, however the answer to it is also very simple: you have to write the testable code from the beginning. Every time the piece of code is done, ask yourself – how will I test it? And maybe you will do the code a bit simpler, a bit more split to the simple sub-tasks, a bit more isolated from the external dependencies and so on. Definitely the testable code writing is a complicated issue and there are a lot of techniques for it: the dependency injection, isolating the business logic from the object instantiation (operator new), using inheritance and polymorphism instead of overly complicated if/switch constructions and so on and so forth.
Of course I have referenced many things from object oriented world which make it easier to use unit testing. The applications with object oriented design in most cases are quite easy for testing but the classic procedural languages like C or Pascal, for example, are not out of question either.
Let’s see how to test the similar example written on ANSI C. Here are your sources:
File: mult.h
#ifndef _MULT_H #define _MULT_H int mult(int a, int b); #endif
File: mult.c
(buggy version)
#include "mult.h" int mult(int a, int b) { int r = 0; if (!a || !b) return 0; if (a == 920 && b == 847) r++; do { if (b & 1) r += a; a <<= 1; } while (b >>= 1); return r; }
I will use another Google testing framework here – cmockery 0.1.2. This framework was designed to test C code and it’s a very powerful framework. On top of just the set of assert_*
functions it can help to find memory leaks and buffer under- and overruns.
Let’s get it:
wget http://cmockery.googlecode.com/files/cmockery-0.1.2.tar.gz
gzip -dc cmockery-0.1.2.tar.gz | tar xvf -
This command will create the cmockery-0.1.2
folder in your current directory. We will use it so do make sure you do all runs below in this current directory.
Let me show you the test suite with the same functionality but written on C:
File: mult_test.h
#ifndef _MULT_TEST_H #define _MULT_TEST_H void mult_simple_test(void **state); void mult_zero_test(void **state); void mult_all_test(void **state); #endif
File: mult_test.c
#include <stdarg.h> #include <stddef.h> #include <setjmp.h> #include <cmockery.h> void mult_simple_test(void **state) { assert_int_equal(91, mult(7, 13)); } void mult_zero_test(void **state) { assert_int_equal(0, mult(0, 7)); assert_int_equal(0, mult(7, 0)); } void mult_all_test(void **state) { int a, b; for (a = 0; a < 1000; ++a) for (b = 0; b < 1000; ++b) assert_int_equal(a * b, mult(a, b)); }
And the runner:
#include <stdarg.h> #include <stddef.h> #include <setjmp.h> #include <cmockery.h> #include "mult_test.h" int main(int argc, char* argv[]) { const UnitTest tests[] = { unit_test(mult_simple_test), unit_test(mult_zero_test), unit_test(mult_all_test), }; return run_tests(tests); }
Let’s compile it with GCC version 3 or higher:
gcc -Icmockery-0.1.2/src/google -o mult_test \
cmockery-0.1.2/src/cmockery.c mult.c mult_test.c runner.c
If everything is correct you should test mult_test
executable. Let’s run it:
./mult_test
and it will print something like this:
mult_simple_test: Starting test
mult_simple_test: Test completed successfully.
mult_zero_test: Starting test
mult_zero_test: Test completed successfully.
mult_all_test: Starting test
0xbe3e8 != 0xbe3e9
ERROR: mult_test.c:19 Failure!
mult_all_test: Test failed.
1 out of 3 tests failed!
mult_all_test
The mult_all_test
fails in the line 19 and it reports that the expected value of multiplication is 0xBE3E8 (decimal 779240 = 920 * 847) but the actual one is 0xBE3E9 (decimal 779240). Now we fix the mult()
function removing buggy line if (a == 920 && b == 847) r++;
:
File: mult.c
(error free version)
#include "mult.h" int mult(int a, int b) { int r = 0; if (!a || !b) return 0; do { if (b & 1) r += a; a <<= 1; } while (b >>= 1); return r; }
and run the test suite again. Now it prints this:
mult_simple_test: Starting test
mult_simple_test: Test completed successfully.
mult_zero_test: Starting test
mult_zero_test: Test completed successfully.
mult_all_test: Starting test
mult_all_test: Test completed successfully.
All 3 tests passed
We see now all three tests work fine. Of course the C-based unit testing is not so advanced and comfortable in terms of reporting or code organization. You have to declare your test cases in the header file and in the runner but this is a limitation of the C language. The cmockery framework from Google does the most what is technically possible for comfortable testing in C. But even if the reporting is not ideal you are always informed about which test fails and in which line.
Other languages have the unit testing frameworks as well. For jUnit for Java, pyUnit for Python and so on. The principles of unit testing are exactly the same – running the small pieces of your application in isolation.
QA (Quality Assurance) testing and regression testing are another big topic. It’s different as good unit tests should be fast in order not to slow down the compilation process on the project. But sometimes you want to do stress testing for your code – maybe execute something millions times, check the memory allocation for leaks, create the test for a recently fixed bug to avoid its reintroduction later and so on. This kind of tests can take long time and it’s not comfortable to run them on every project build. Here the QA and regression testing step onto the scene. It’s also quite an interesting topic and I will try to cover it soon as well.
vs_double_semicolumn.c
):
void main() { int a;; int b; }
Compile (in С mode, there is no /TP
):
cl vs_double_semicolumn.c
The result:
vs_double_semicolumn.c
vs_double_semicolumn.c(3) : error C2143: syntax error : missing ';' before 'type'
The result of Codegear/Borland is roughly the same (though the error message is more clear):
CodeGear C++ 5.93 for Win32 Copyright (c) 1993, 2007 CodeGear
vs_double_semicolumn.c:
Error E2140 vs_double_semicolumn.c 3: Declaration is not allowed here in function main
*** 1 errors in Compile ***
The problem hides behind an accidental typo of the doubled ;
character. By the way, this example is absolutely real from life. Just one misprint raises a lot of questions.
It turns out that the second ;
character is treated as an empty statement but not as an empty variable declaration. The compiler decides that declarations of variables are finished and a block of statements begins, therefore it reasonably complains on the b
declaration expecting statements.
I’ve checked it on gcc and native compilers of AIX, Solaris and HP-UX. All of those have eaten this example without any problems.
Related posts:
bcc32_5.93_cast_bug.cpp
):
class A {}; class C {}; A* a; A* b = static_cast<C*>(a);
When using bcc32.exe
(version 5.93) from the Codegear RAD Studio 2007:
bcc32 -c bcc32_5.93_cast_bug.cpp
It dies with an internal compiler error:
CodeGear C++ 5.93 for Win32 Copyright (c) 1993, 2007 CodeGear
bcc32_5.93_cast_bug.cpp:
Fatal F1004 bcc32_5.93_cast_bug.cpp 4: Internal compiler error at 0x44b34e with base 0x400000
Fatal F1004 bcc32_5.93_cast_bug.cpp 4: Internal compiler error
Nice one. I love internal compiler errors. Do you have some similar stuff in your collection?
virtual_funct_const.cpp
):
#include <iostream> class A { public: A() { construct(); } ~A() { destruct(); } virtual void construct() { std::cout << "A::construct()" << std::endl; } virtual void destruct() { std::cout << "A::destruct()" << std::endl; } }; class B: public A { public: B() { construct(); } ~B() { destruct(); } virtual void construct() { std::cout << "B::construct()" << std::endl; } virtual void destruct() { std::cout << "B::destruct()" << std::endl; } }; int main() { B b; return 0; }
What does this program print out?
Here we go:
A::construct()
B::construct()
B::destruct()
A::destruct()
Seems that the constructors and destructors of the classes A
and B
called the functions from its own class even if those functions construct()
and destruct()
were declared as virtual.
There is no magic here but the rule: a virtual function becomes non-virtual when called from a constructor or a destructor.
Any rule has to be memorized which is not convenient. It’s much better to just understand why it works this way. This behaviour is based on the main principle of the inheritance implementation in C++: when an object is being constructed, its constructors are called in order starting from the base class of the hierarchy to the last inherited one. Destructors are called in the reversed order.
A constructor always works with the assumption that its child classes are not constructed yet, that is why there is no way to call anything declared in the child classes. Therefore when calling a virtual function, the constructor has nothing else to do but to call its own implementation of that function. It seems the mechanism of virtual functions doesn’t work here. It really doesn’t because the virtual functions table of the child class doesn’t override the current table yet.
For a destructor everything works vice versa. A destructor knows that all child classes are already destructed and nothing can be called from them. Therefore it just changes an address of the virtual functions table to the address of its own table address and nicely calls an implementation of the virtual function defined in its own class.
Thus a virtual function is not virtual if it’s called from a constructor or a destructor.
The traditional conclusion: Good algorithms are better than supercomputers.
Finally my rank is 2896 (nickname ‘begoon’) from all roughly 9700 competitors, almost closer to a pensionary level, and the score is 76.
It’s great that there are three Russian flags in the top twenty.
They say it’s enough to get only 33 points for the Round 1 which means you had to solve one small dataset and one large dataset from any problem.
Does anybody participate?
std::string
. Is it worth using it at all or our own string class implementation could be better?
Amusingly, the majority of people, which I asked to sketch out more or less efficient implementation of the string class, wrote roughly the following:
class String { public: explicit String(const std::string& value) { init(value.c_str(), value.length()); } String(const String& value) { init(value.data_, value.sz_); } ~String() { free(data_); } String& operator=(const String& value) { if (this != &value) { if (value.sz_ > sz_) data_ = (char*)std::realloc(data_, value.sz_); sz_ = value.sz_; std::memcpy(data_, value.data_, sz_); } return *this; } private: void init(const char* data, size_t sz) { sz_ = sz; data_ = (char*)malloc(sz_); std::memcpy(data_, data, sz_); } char* data_; size_t sz_; };
Obviously with such implementation of the assignment operator the string can only increase memory utilisation, not reduce. It’s done deliberately to save some extra time on assigning.
Practically nobody thought immediately about a moving operation, for example, a swap. For some reason the presence of a copy constructor and an assignment operator was considering to be sufficient.
I’ve written a test. The test sorts an array of long strings. The strings are represented in four ways: a std::string
object, a std::string
pointer, an object of my homemade String
class (see above) and a pointer to String
.
Apparently, use of the pointer should be the most efficient method because in this case std::sort()
swaps only pointers but not objects.
But it would be interesting to compare how my simple implementation performs against std::string
.
So, std_string.cpp
:
#include <iostream> #include <sstream> #include <string> #include <vector> #include <algorithm> #include <cstdlib> #include <cstring> #include <cassert> #include "gtest/gtest.h" static const int N = 100; // This homemade class implements the more or less efficient // string in terms of copying. class String { public: // "explicit" disables any implicit cast making sure // which constructor exactly is being called. explicit String(const std::string& value) { init(value.c_str(), value.length()); } String(const String& value) { init(value.data_, value.sz_); } ~String() { free(data_); } // Perhaps this method is only one attempt to use memory allocation // efficiently. String& operator=(const String& value) { if (this != &value) { // Memory is re-allocated only if a source is longer the current // string. It's clear that this implementation will only increase // memory allocated by the string. if (value.sz_ > sz_) data_ = (char*)std::realloc(data_, value.sz_); sz_ = value.sz_; std::memcpy(data_, value.data_, sz_); } return *this; } friend class StringCmp; friend class StringPointerCmp; private: void init(const char* data, size_t sz) { sz_ = sz; data_ = (char*)malloc(sz_); std::memcpy(data_, data, sz_); } char* data_; size_t sz_; }; std::vector<std::string> std_strings; std::vector<std::string*> std_strings_p; std::vector<String> strings; std::vector<String*> strings_p; // Functor to compare two std::string. class StlStringCmp { public: bool operator()(const std::string& a, const std::string& b) { return a < b; } }; TEST(SortingStlString, StlString) { std::sort(std_strings.begin(), std_strings.end(), StlStringCmp()); } // Functor to compare two std::string*. class StlStringPointerCmp { public: bool operator()(const std::string* a, const std::string* b) { return *a < *b; } }; TEST(SortingStlString, StlStringPointer) { std::sort(std_strings_p.begin(), std_strings_p.end(), StlStringPointerCmp()); } // Functor to compare two String. class StringCmp { public: bool operator()(const String& a, const String& b) { assert(a.sz_ == b.sz_); return std::memcmp(a.data_, b.data_, a.sz_); } }; TEST(SortingStlString, String) { std::sort(strings.begin(), strings.end(), StringCmp()); } // Functor to compare two String*. class StringPointerCmp { public: bool operator()(const String* a, const String* b) { assert(a->sz_ == b->sz_); return std::memcmp(a->data_, b->data_, a->sz_); } }; TEST(SortingStlString, StringPointer) { std::sort(strings_p.begin(), strings_p.end(), StringPointerCmp()); } int main(int argc, char* argv[]) { // The filler to make strings long enough making their copying expensive. std::string big(1024 * 1024, '?'); for (int i = 0; i < N; ++i) { // All strings are the same length. The comparison functions rely on it. std::stringstream fmt; fmt << N * 2 - i << big; std_strings.push_back(fmt.str()); std_strings_p.push_back(new std::string(fmt.str())); strings.push_back(String(fmt.str())); strings_p.push_back(new String(fmt.str())); } testing::InitGoogleTest(&argc, argv); // Enforce to print out tests' timing. testing::GTEST_FLAG(print_time) = true; return RUN_ALL_TESTS(); }
Compile:
cl /O2 /EHsc /I. std_string.cpp gtest-all.cc
Run:
[==========] Running 4 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 4 tests from SortingStlString
[ RUN ] SortingStlString.StlString
[ OK ] SortingStlString.StlString (203 ms)
[ RUN ] SortingStlString.StlStringPointer
[ OK ] SortingStlString.StlStringPointer (0 ms)
[ RUN ] SortingStlString.String
[ OK ] SortingStlString.String (891 ms)
[ RUN ] SortingStlString.StringPointer
[ OK ] SortingStlString.StringPointer (0 ms)
[----------] 4 tests from SortingStlString (1125 ms total)
[----------] Global test environment tear-down
[==========] 4 tests from 1 test case ran. (1125 ms total)
[ PASSED ] 4 tests.
Apparently, the tests using pointers work equally fast but when the objects are in use, std::string
has overrun my homemade implementation 4 times - 203 ms vs 891 ms.
It’s simply to figure out why. To swap elements std::sort()
uses the template function std::swap()
which exchanges objects without physical copying the string contents.
Eventually I’ve convinced myself that in most cases std::string
solves all problems. But what about adding more functionality to the std::string
? For instance, a word search.
The problem is that the destructor of std::string
isn’t virtual (maybe from considerations of efficiency), but an inheritance from a class with the non-virtual destructor isn’t right undertaking in C++.
The STL author, Alexander Stepanov, in his article Notes for the Programming course at Adobe, advises to implement additional functionality for standard STL containers as template algorithms. There are many advantages doing so, for example, any string parser implemented via iterators as a template algorithm becomes usable automatically for all other containers having the same iterators.
Interestingly what Stepanov says about the length()
function of STL containers (in the article above he describes the process of creation the efficient container step by step):
While we could make a member function to return length, it is better to make it a global friend function. If we do that, we will be able eventually to define the same function to work on built-in arrays and achieve greater uniformity of design. I made size into a member function in STL in an attempt to please the standard committee. I knew that begin, end and size should be global functions but was not willing to risk another fight with the committee.
Alexander Stepanov
Summarizing, it’s worth to just trust std::string
. For majority of problems it performs pretty well, and its functionality can be extended by template algorithms.
Follow the announcements.