The black box optimization competition challenges its participants to achieve as good as possible solution to unknown optimization problems through a black-box interface within a given budget of objective function evaluations.
The competition is organized into tracks, which again consist of problems. Each problem is defined by the dimension d of its search space Ω, which for all problems takes the form Ω = [0,1]d. For each problem the participant has a predefined budget B of evaluations f(x) of an unknown function f at his disposal. Hence f can be evaluated in B different points x ∈ Ω. The goal is to obtain an as low as possible value f(x) for the best point found (single-objective case), or to dominate the largest possible share of the objective space if f is vector-valued (multi-objective optimization). This setting is known as black box optimization.
Each participant is identified by an account (user name), which is protected with a password. User accounts are created by the competition organizers on request. Write to firstname.lastname@example.org and be sure to include your name, affiliation and your preferred user name. Accounts can be managed through a web interface, which is accessible through the login box on the top right of the main page.
Besides the actual competition track there exists special tracks named "trial" and "trialMO". These tracks consist of dummy problems which are, in number and dimension (but not necessarily in their other characteristics), comparable to competition tracks. These tracks should be used to thoroughly test optimization software before switching to a competition track. The special property of the trial tracks is that the optimization can be repeated many times, since the number of objective function evaluations can be reset through the web interface. Note that a user can solve each problem in a competition track only once. Hence, it must be made double sure on the trial tracks that the optimization software works exactly as expected.
The web interface offers the following functionality:
- Participants can manage their data, such as name, professional affiliation, and email address. They can also change their password.
- Participants can upload material on their algorithms, such as scientific papers and source code. This is entirely optional, however, very much recommended in the spirit of open and reproducible research. However, participation with closed source algorithms is explicitly allowed.
The trial and trialMO tracks can be reset. This action resets the number
of evaluations for all problems in the "trial" track to zero.
Afterwards optimization can start over, which is a prerequisite for
efficient debugging of user code.
Note:This action resets only the data stored on the server. At the same time participants should remove all log files related to the track, e.g., with a Unix shell command such as:rm -f ./logfile_[accountname]_trial_*.logrm -f ./logfile_[accountname]_trialMO_*.log
- The optimization progress can be monitored. Participants may at any time check the progress of their algorithm. This is useful for monitoring optimization runs in a distributed computing environment such as a cluster.
The overall software architecture of the black box optimization competition is based on a client/server model. On the hardware level there exists a server machine that collects data, while the actual optimization tasks are running on client machines at the participant's side. This means in particular that the competition does not provide computing resources to participants. Instead participants use any computing resources at their disposal to participate in the competition. The only requirement is that clients are connected to the internet so that results can be communicated to the competition server.
On the software level participants do not get in touch with the competition server directly. Instead their optimization software talks to the server through a dynamic library (dll on windows, so (shared object) on linux, and dylib on MacOS). This library is provided as a binary file. It exports a well-defined interface, and otherwise truly is a black box to participants. Its interface functions are explained in detail in the next section. The maybe most important of these is the evaluate(...) function, which is the actual black box optimization interface.
Participants are expected to execute their preferred black box optimization algorithm on each problem in a competition track. However, the program implementing the algorithm must organize a few things before it can get started, e.g., log into the system, select a problem for optimization, perform proper error handling, etc. This program can be written in any programming language capable of invoking the dynamic library or opening a TCP/IP socket connection. Example programs (implementing random search and the classic downhill simplex method) are provided in C, C++, Matlab/Octave, Python, and Java. Socket-based access requires a standalone proxy program that comes with the competition software collection. It allows to access all library functions transparently with an ASCII-based protocol through the socket connection, however, with increased latency as compared to direct library calls (described below).
Dynamic Library Setup
The client (user) program most connect to the dynamic link library that is at the heart of the black box optimization competition software. The library is interfaced in plain C. It consists of two files, the actual library (bbcomp.dll on Windows, libbbcomp.so on Linux, and libbbcomp.dylib on MacOS), and an accompanying header file bbcomplib.h. The easiest way to learn how to connect to this library is to have a look at the example clients (included in the software). The client's functionality is explained below.
The following additional information may be useful, e.g., when accessing the library from a programming language not covered by the examples. In most of these cases you should get away with writing minimal glue code, probably at least partly written in C. The header file bbcomplib.h provides four macros: DLL_DECLARATIONS, DLL_LOAD, DLL_LOAD_STATUS, and DLL_CLOSE. They are supposed to be used as follows:#include <bbcomplib.h>
if (DLL_LOAD_STATUS < 0)
printf("error loading library\n");
...your actual code goes here...
After DLL_LOAD and before DLL_CLOSE, and as long as DLL_DECLARATIONS is in scope, the functions exported by the dynamic library can be called. What happens behind the scenes is that DLL_DECLARATIONS declares (global) variables, which happen to be function pointers to the library's interface functions. DLL_LOAD initialized these pointers. The DLL_LOAD_STATUS variable contains a negative value after DLL_LOAD if initialization failed, and a positive value otherwise. DLL_CLOSE performs cleanup operations.
This section acts as a reference documentation for the library. It exports a plain C-language interface. The interface should hence be accessible from many programming languages.
Note:The library is stateful and not thread safe. Hence it cannot be used by more than one optimizer running in one thread per process at a time. However, it can be accessed by multiple processes per machine and even from multiple machines, which allows for massive parallelism. We recommend users to limit themselves to at most 10 simultaneous connections.
int configure(int history, const char* logfilepath);
Set global configuration options. This call is optional, however, it can only occur before login.
- history: Setting this value to zero disables the history, any other value enables the history (the default). Enabling the history has the effect that each single function evaluation is logged to local files. This means that this information is available even after a program crash or power outage, possibly allowing an implementation to recover gracefully and continue the optimization run. The downside is that (depending on the size of the track) the files may grow as large as 30 GB, which may be an issue in some restricted environments. Also, the history information may not be of value to all optimizers. This is why it can be turned off.
- logfilepath: This path is prepended to file names of log files. This gives the user control over where to store these files. The default value is ".", referring to the current directory.
Note: It is strongly recommend to leave logging enabled if possible. The logging mechanism safeguards against failures of all kinds, such as network problems, power outage, program crashes, and many more. Recall once more that you have only a single trial.
int login(const char* username, const char* password);
Login to the system. This call results in communication with the competition server. It validates the user's credentials. Multiple concurrent logins with the same credentials are possible to allow for parallel evaluation of problems on different cores/machines, but not from multiple threads within the same process. The account becomes a persistent property of the library state; subsequent function calls refer to this account. The function returns 0 on error.
Return the number N of tracks available to the user. Valid track indices range from 0 to N-1. The function returns 0 on error.
const char* trackName(int trackindex);
Return the name of the track. Tracks are identified by their name, not by their index (which may change depending on user rights). The function returns a NULL pointer on error.
int setTrack(const char* trackname);
Select a track. Subsequent function calls refer to this track. The function returns 0 on error.
Return the number of problems M in the current track. Valid problem IDs range from 0 to M-1. The function returns 0 on error.
int setProblem(int problemID);
Select a problem within the chosen track for optimization. Subsequent function calls refer to this problem. The function returns 0 on error.
Return the dimension of the search space of the current problem. The function returns 0 on error.
Return the number of objectives, i.e., the number of real values returned by the evaluate function. For (standard) single-objective optimization this function always returns 1. For multi-objective optimization this function returns a value greater than one. The function returns 0 on error.
Return the evaluation budget for the current problem. This is the total available budget, not the remaining budget (which can be obtained by subtracting the result of the evaluations function). Calling evaluate more than this number of times on the current problem will fail. The function returns 0 on error.
Return the number of evaluations so far for the current problem. The function returns -1 on error.
int evaluate(double* point, double* value);
Evaluate the objective function.
- point: Pointer to a linear memory array of double numbers. The size of the array is expected to coincide with the return value of the dimension function.
- value: Pointer to a double number (single-objective case), or to a linear memory array of double numbers (multi-objective case). The size of the array is expected to coincide with the return value of the numberOfObjectives function. On success, the value (number or array) is filled in with the value(s) of the objective function(s) at the given point.
Note:This function fails if the predefined budget of function evaluations is used up.
Note:It is assumed that all variables are within the unit hypercube, i.e., the box constraint
int history(int index, double* point, double* value);
Recover a previous evaluation from the log file. Note: if logging was disabled through configure or if the log file was removed or corrupted then this function will fail.
- index: Time index of the evaluation. If T queries to evaluate were already made then the index can be any number in the range 0 to T-1.
- point: Pointer to a linear memory array of double numbers. The size of the array is expected to coincide with the return value of the dimension function. Memory management is up to the caller, i.e., the memory must be allocated before the call and freed afterwards.
- value: Pointer to a double number (single-objective case), or to a linear memory array of double numbers (multi-objective case). The size of the array is expected to coincide with the return value of the numberOfObjectives function. The value (number or array) is filled in with the value(s) of the objective function(s). Memory management is up to the caller, i.e., the memory must be allocated before the call and freed afterwards.
const char* errorMessage();
This function returns a human readable error message for the previous error.
The Socket-based Proxy
In some programming environments it is non-trivial to access a dynamic library and its C-language interface. In this case an alternative access method is offered through a standalone executable acting as a proxy for the dynamic library. This proxy forwards the C-language interface to a client through a socket connection. This approach is used, e.g., for the Java frontend. The default port is 7000, which can be changed through a command line option so that multiple proxies can run on a single machine.
The protocol is line-based. A line is defined as a sequence of ASCII characters in the range 32 to 126, ending with code 10 (CR). The sequence 13+10 (LF+CR) is not recognized and results in an error. Each line received by the proxy is interpreted as a function call in the formatfunctionname(param1,param2,...,paramN)
There are four data types:
- integers are decimal numbers as recognized by the C standard library function strtol.
- strings are enclosed in double quotes "...". There is no escaping mechanism, hence double quotes cannot appear inside a string.
- floating point numbers are decimal numbers as recognized by the C standard library function strtod.
- vectors (of floating point numbers) are enclosed in square brackets [...] with components separated with spaces (not commas).
sender: evaluate([0.92241 0.1817 0.76203])
proxy: 1,[0.92241 0.1817 0.76203],3.4419
proxy: "index out of range"
proxy: ERROR,"unknown function 'cheat'"
proxy: ERROR,"failed to parse request parameters"
Note the difference between the evaluate library function and this example. The value parameter is actually an output of the function, hence it is not passed as a parameter but rather is returned as a second value after the success indicator return code "1". A similar consideration applies to the history function. Its first parameter is the time index. The parameters point and value are return values, hence they are not passed as parameters.
In case of a protocol violation the proxy returns the special sequenceERROR,"this is a human readable error message" .
Note that this mechanism does not replace the role of the errorMessage function. It is only applied to communication errors between client and proxy.
This section describes the working principle of the example clients (available in various programming languages) and how they invoke the library functions.
Note:All example clients should run out-of-the-box. They use a demonstration account called demoaccount. Note that this account cannot participate in the competition. It is limited to the "trial" and "trialMO" tracks. Error messages are expected if the client is run before the corresponding track opens or after it closes.
The example clients are intended to serve as templates and starting points for the implementation of actual competition entries. The example clients in all language execute the same sequence of commands. This sequence is discussed in the sequel for the dummy clients performing pure random search.
The first action is to call the configure function. This step is optional. It allows to define settings related to the evaluation history.
The first essential step is the login call. Simply provide you username and password to this function. It makes the dynamic library setup a socket connection to the competition server. This connection will remain open for the duration of the use of the library by the present process. The participant remains logged in for this duration, and all subsequent calls implicitly refer to the logged in participant. Multiple such connections can coexist per participant so that one participant can use multiple cores to solve problems in parallel.
A track is selected with the setTrack function. The track can be either hard-coded or obtained from the above list.
The function numberOfProblems returns the number of problems in the selected track.
A problem is selected with the setProblem function. This problem is now ready for optimization.
The functions dimension, numberOfObjectives, and budget return properties of the selected problem, while evaluations returns the number of evaluations of the objective function on this problem so far.
Once these problem characteristics have been obtained the demonstration program starts an extremely simple optimization strategy: random search with i.i.d. uniform sampling. Each attempt to solve the problem consists of sampling a search point x uniformly at random from the search space. This point is then evaluated by calling evaluate.
Now assume that the client program has crashed, say, due to a power outage. Then it would be fatal to lose the evaluations so far, since they are limited by a budget. The library helps out in this case by providing the history of previous evaluations at any time (unless this feature has been disabled by a call to configure, see above). The demo client program implements an extremely simple recovery strategy: when it detects that the optimization was run partially then instead of drawing samples it requests historic samples through the history function. This allows the algorithm to repeat its steps and to continue from exactly where it was at the time of the program crash. Of course, since pure random search is stateless there is no state to recover, however, the example program demonstrates the technique in principle by outputting newly obtained as well as historic samples.
After the optimization loop the program checks whether the budget was indeed used up. This check is only for demonstration purposes.
Finally the program terminates, which results in closing the connection to the competition server. Alternatively the program could continue to optimize other problems or even problems from other tracks. A typical setup is to add a loop over all problems of a track. Alternatively this loop can be external to the program, e.g., to take advantage of parallel hardware. However, for parallel processing take this note into account.
The following problems and bugs are known. All known bugs are fixed in the most recent version of the software, hence this list is mostly for historical completeness. If you encounter any of these problems then be sure to update to the most recent version of the BBComp software.
- Version 1.2.0 was not running the GVGPC track correctly on Windows due to odd behavior of the JVM. Version 1.2.1 should fix this problem.
- Version 1.1.1 of the Windows version contained a but that prevented connecting to the competition server. The bug should be fixed in version 1.1.2.
- Some firewalls block port 6881, which was used for communication with the server up to version 1.0.5 of the client software. The new port 39772 was opened on 06.03.2015. It is used from client software versions 1.0.6 onwards. The old port is still available.
- Up to versions 1.0.4 the Windows libraries for the Matlab client were messed up. The bugs were fixed in version 1.0.5.
- In versions 1.0.3 the advanced Python client had bugs evaluating error messages. The bugs were fixed in version 1.0.4.
- Under certain circumstances, versions 1.0.1 and 1.0.2 of the library did not keep their internal state consistent. The only solution was to unload and reload the library. The bugs were fixed in version 1.0.3.
- In version 1.0.0 the Java client contained a few subtle bugs. The bugs were fixed in version 1.0.1. Check your version.txt file and make sure you use the most recent version.
- The downloads released on 30.11.2014 did not include version information. Fix: The file version.txt was added to the package in version 1.0.1.