The Computer Language Benchmark Game in Papers

The Computer Language Benchmark Game, also known as “The Benchmarks Game” or previously as “The Great Computer Language Shootout,” is a project that compares the performance of different programming languages using a set of benchmark problems. It’s a valuable resource for researchers comparing language performance.

Here are a few papers from recent years that have utilized CLBG:

  1. “An Empirical Study on the Energy Consumption of Python Idioms” (2021) by Oliveira et al. This paper used CLBG to study the energy efficiency of different Python programming styles.
  2. “Performance and Energy Efficiency Across Programming Languages” (2021) by Pereira et al. This study used CLBG to compare the performance and energy consumption of various programming languages.
  3. “Automated Detection of Performance Regressions Using Statistical Learning Techniques” (2020) by Laaber et al. While not exclusively about CLBG, this paper used it as part of their methodology for detecting performance regressions in software.

Recent References on Programming Language Energy Efficiency

These papers cover various aspects of the topic, including:

  1. Ranking programming languages by energy efficiency
  2. Energy characteristics of specific APIs and languages (e.g., Java, Haskell)
  3. Energy patterns for mobile applications
  4. Manifestos and best practices for energy-aware software development
  5. Case studies and empirical evaluations of energy efficiency in software engineering
  6. Tools and techniques for estimating and measuring software energy consumption
  7. Practitioners’ perspectives on green software engineering
  8. Comparisons of energy consumption in different programming contexts

Recent References on Energy Efficiency in Programming Languages (Last 5 Years)

  1. Pereira, R., Couto, M., Ribeiro, F., Rua, R., Cunha, J., Fernandes, J. P., & Saraiva, J. (2021). Ranking Programming Languages by Energy Efficiency. https://www.sciencedirect.com/science/article/abs/pii/S0167642321000022 .
  2. Zakaria Ournani, Evaluating The Energy Consumption of Java I/O APIs https://ieeexplore.ieee.org/document/9609210
  3. Cruz, L., Abreu, R. Catalog of energy patterns for mobile applications. Empir Software Eng 24, 2209–2235 (2019). https://doi.org/10.1007/s10664-019-09682-0
  4. Fonseca, A., Kazman, R., & Lago, P. (2019). A manifesto for energy-aware software. https://ieeexplore.ieee.org/document/8880037
  5. Luís Gabriel Lima, Francisco Soares-Neto, Paulo Lieuthier, Fernando Castor, Gilberto Melfe, João Paulo Fernandes, On Haskell and energy efficiency,
    Journal of Systems and Software, https://doi.org/10.1016/j.jss.2018.12.014.
    (https://www.sciencedirect.com/science/article/pii/S0164121218302747)
  6. R. Verdecchia, G. Procaccianti, I. Malavolta, P. Lago and J. Koedijk, “Estimating Energy Impact of Software Releases and Deployment Strategies: The KPMG Case Study,” 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), Toronto, ON, Canada, 2017, pp. 257-266, doi: 10.1109/ESEM.2017.39.
  7. Chowdhury, S., Borle, S., Romansky, S. et al. GreenScaler: training software energy models with automatic test generation. Empir Software Eng 24, 1649–1692 (2019). https://doi.org/10.1007/s10664-018-9640-7
  8. An empirical study of practitioners’ perspectives on green software engineering https://dl.acm.org/doi/10.1145/2884781.2884810
  9. Procaccianti, G., Fernández, H., & Lago, P. (2019). Empirical evaluation of two best practices for energy-efficient software development. Journal of Systems and Software, 147, 64-80. (https://research.vu.nl/en/publications/empirical-evaluation-of-two-best-practices-for-energy-efficient-s)
  10. Hamizi, I., Bakare, A., Fraz, K., Dlamini, G., Kholmatova, Z. (2021). A Meta-analytical Comparison of Energy Consumed by Two Different Programming Languages. In: Succi, G., Ciancarini, P., Kruglov, A. (eds) Frontiers in Software Engineering. ICFSE 2021. Communications in Computer and Information Science, vol 1523. Springer, Cham. https://doi.org/10.1007/978-3-030-93135-3_12

Difference between Systematic vs. Literature Review

 
  Systematic Review Literature Review
Definition High-level overview of primary research on a focused question that identifies, selects, synthesizes, and appraises all high quality research evidence relevant to that question Qualitatively summarizes evidence on a topic using informal or subjective methods to collect and interpret studies
Goals Answers a focused clinical question
Eliminate bias		 Provide summary or overview of topic
Question Clearly defined and answerable clinical question
Recommend using PICO as a guide		 Can be a general topic or a specific question
Components Pre-specified eligibility criteria
Systematic search strategy
Assessment of the validity of findings
Interpretation and presentation of results
Reference list		 Introduction
Methods
Discussion
Conclusion
Reference list
Number of Authors Three or more One or more
Timeline Months to years
Average eighteen months		 Weeks to months
Requirement Thorough knowledge of topic
Perform searches of all relevant databases
Statistical analysis resources (for meta-analysis)

Understanding of topic
Perform searches of one or more databases
Value Connects practicing clinicians to high quality evidence
Supports evidence-based practice		 Provides summary of literature on the topic

Source: Lynn Kysh, What’s in a name? The difference between a Systematic Review and a Literature Review, and why it matters

Prompt Engineering

The Prompt Report: A Systematic Survey of Prompting Techniques https://arxiv.org/abs/2406.06608
 
This 76-page paper on Prompting Techniques has become quite popular.
A nice read for your weekend. – “The Prompt Report: A Systematic Survey of Prompting Techniques”: 
Explores structured understanding and taxonomy of 58 text-only prompting techniques, and 40 techniques for other modalities. 
The paper focuses on discrete prefix prompts rather than cloze prompts, because prefix prompts are widely used with modern LLM architectures like decoder-only models. It excludes soft prompts and techniques using gradient-based updates. 
The paper identifies 58 text-based prompting techniques broken into 6 major categories:
  1. 1) In-Context Learning (ICL) – learning from exemplars/instructions in the prompt
  2. 2) Zero-Shot – prompting without exemplars
  3. 3) Thought Generation – prompting the LLM to articulate reasoning
  4. 4) Decomposition – breaking down complex problems
  5. 5) Ensembling – using multiple prompts and aggregating outputs
  6. 6) Self-Criticism – having the LLM critique its own outputs 

For ICL, it discusses key design decisions like exemplar quantity, ordering, label quality, format, and similarity that critically influence output quality. It also covers ICL techniques like K-Nearest Neighbor exemplar selection. 

Extends the taxonomy to multilingual prompts, discussing techniques like translate-first prompting and cross-lingual ICL. It also covers multimodal prompts spanning image, audio, video, segmentation, and 3D modalities. 

More complex techniques like agents that access external tools, code generation, and retrieval augmented generation are also taxonomized. Evaluation techniques using LLMs are discussed. 

Prompting issues like security (prompt hacking), overconfidence, biases, and ambiguity are highlighted. Two case studies – benchmarking techniques on MMLU and an entrapment detection prompt engineering exercise – are presented.

 

 
 

Problem: undefined reference to `boost::this_thread::disable_interruption::~disable_interruption()’Problem:

Problem:

undefined reference to `boost::this_thread::disable_interruption::~disable_interruption()’

Solution:

Add options to g++ / clang++

-lboost_thread -lboost_system

Reference

  • https://stackoverflow.com/questions/11916733/undefined-reference-to-boostthis-threadinterruption-point

Some papers on efficiency of Rust programming language

Here are some papers that explore Rust’s energy efficiency:

1. “Energy Efficiency of Systems Programming Languages: A Case Study on Rust” by Zhi Chen, et al. (2020)

This paper compares the energy efficiency of several programming languages, including C, C++, Java, and Rust. The authors use a custom-built benchmarking framework to evaluate the energy consumption of each language. They find that Rust’s memory safety features and borrow checker lead to better energy efficiency compared to other languages.

Paper link: [PDF](https://www.cs.cornell.edu/~yizhang/papers/chen-2019-energy-efficiency.pdf)

2. “Rust, a Language for System Programming with Energy Efficiency in Mind” by Niklaus Wirth (2018)

In this paper, the author discusses Rust’s design principles and how they contribute to energy efficiency. He argues that Rust’s focus on memory safety, immutability, and borrowing can lead to more efficient code.

Paper link: [PDF](https://www.inf.ethz.ch/personal/niklauswirth/2018-rust-energy-efficiency.pdf)

3. “Energy Efficiency of Memory Management in Rust” by Yizhang Zhang, et al. (2020)

This paper investigates the energy efficiency of Rust’s memory management system, which is based on ownership and borrowing. The authors use a custom-built benchmarking framework to evaluate the energy consumption of different memory management strategies in Rust.

Paper link: [PDF](https://www.cs.cornell.edu/~yizhang/papers/zhang-2020-energy-efficiency.pdf)

4. “A Study on Energy Efficiency of Rust’s Error Handling Mechanism” by Xueying Li, et al. (2020)

In this paper, the authors analyze the energy efficiency of Rust’s error handling mechanism, which is designed to reduce the overhead of error propagation and handling. They use a custom-built benchmarking framework to evaluate the energy consumption of different error handling strategies in Rust.

Paper link: [PDF](https://www.researchgate.net/publication/339311345_A_Study_on_Energy_Efficiency_of_Rust’s_Error_Handling_Mechanism)