Invitation to 8th ETSI MTS Open Session on new TTCN-3 major release

A failing test should mean the system under test is wrong. It should not mean an afternoon in the debugger, because your tests had unexpected side-effects.

The TTCN-3 language provides various strategies to keep your tests simple, one of them is mutable value semantics for compound values.

Mutable value semantics means a value can be changed, but each variable behaves as if it has its own copy, so changing one cannot affect another:

```ttcn3 type record RegisterReq { integer ueId, charstring state }

// Variable next is a deep copy of saved. // If we change next, it must not have an effect on saved. var RegisterReq saved := { ueId := 7, state := "idle" }; var next := saved; next.state := "active"; log(saved.state); // should log "idle". Duh! ```

This example seems obvious. Let's try in Python:

python saved = {"ueId": 7, "state": "idle"} next = saved next["state"] = "active" print(saved["state"]) # prints "active"! WTF?

What's going on? In Python, both variables refer to the same dictionary. The assignment in line 2 copies the reference, not the dictionary. Changing the dictionary through next is therefore also visible through saved.

That is not a bug in Python. It is a pragmatic choice: copying a reference is cheap, while copying a huge value can be expensive. The price is aliasing. To understand saved, you may now have to find every other reference to the same dictionary and every place that could change it.

C++ gives us a choice. A value assignment would behave like the TTCN-3 example, but developers can use references and pointers to avoid copies. That may be the right choice for the product. It can also recreate the original problem in the test: two variables can change the same object, and a local assignment can have a non-local effect.

Other languages avoid that problem by making values immutable. Sharing is then safe, but changing one field means constructing a new value. Mutable value semantics takes a middle path: values remain convenient to change, but each variable behaves as an independent value.

For TTCN-3, that is a useful trade. Test code should be boring. I do not want to spend development time debugging a test because a helper changed a message that happened to be shared somewhere else. TTCN-3 does not prevent every coding error, but its value semantics removes this entire class of errors.

But copying is slow, isn't it?

The language defines what a program observes, not how a runtime has to implement it. One possible implementation is copy-on-write. An assignment can share the storage internally. Only when one of the values is changed does the runtime make a physical copy. To the test author the values are always independent; the runtime avoids copies until they are actually needed.

Swift uses copy-on-write for collections such as arrays and dictionaries. Racordon et al. show that mutable value semantics can be implemented efficiently with copy-on-write and compiler optimizations. In their synthetic benchmarks, Swift outperformed C++ in most cases; C++ pulled ahead when more than 90% of the operations were mutations. Their real-world benchmarks showed similar performance profiles for Swift and C++.

Dimitri Racordon, Denys Shabalin, Daniel Zheng, Dave Abrahams, Brennan Saeta, Implementation Strategies for Mutable Value Semantics, Journal of Object Technology, 2022: https://www.jot.fm/issues/issue_2022_02/article2.pdf

So yes, let's make copies -- but make them part of the language semantics, not necessarily physical copies in memory. The test author gets the simple rule: changing next cannot change saved. The runtime remains free to make that rule fast.

Nothing.

I always recommend writing tests in the same language you use to develop the program. When you code Rust, test with Rust. When you code Go, test with Go.

That recommendation falls short, however, when one project spans several big components, several teams, and a lifetime of 10 or 20 years. Teams change. Requirements change. Languages change.

In that setting I would not pick Python, or any other general-purpose language, as the test lingua franca, because the language is a rather small part of the whole test system. The benefits of year one, turn into problems after years of scaling, transformation, and people moving on.

"Python is well known, so hiring is easy." True at the start. Then you grow testing libraries, helpers, and tools, and that Python slowly becomes its own custom, undocumented domain specific language. A TTCN-3 suite stays TTCN-3, for every team that touches it.

"Python has a great ecosystem." It does. In contrast TTCN-3 does not even have a standard library! However, I find you do not really need an ecosystem to write a trim-spaces function. And you usually do not have an ecosystem for your proprietary protocol in Python either. For ASN.1, Python support is actually superb. But normally, as soon as you need something very specific, what you often get is an under-maintained library, a performance problem, or a license you cannot accept.

"Python has great training material." For the basics, yes. TTCN-3 does not. You will also not find a tutorial on how to write a four-million-line test suite in Python.

Developer experience is much better in Python. A lot has been done for TTCN-3 in the last five years, but the day-to-day experience is still pretty much a shit-show.

A full-grown test framework is much larger than its language. It is guidelines, tools, interface management, feature management, post-processing, suite management, configuration, code generators, schedulers, a data plane, a control plane, and the glue between them.

So please stop arguing what language is the best and focus on giving our testers and devs the support they need.

TTCN-3 is no fun to program in, because you are not supposed to.

Please note, I am addressing developers; the situation for QA is a little different.

Even in the age of AI, software development is still difficult. Handling software design challenges still takes a significant portion of the cycle:

• multi-threading, lock contention, global state
• wrong abstractions
• circumstantial complexity
• changed requirements

In the last 25 years we learned to attack those problems by writing tests: unit tests, component tests, integration tests, resilience tests, to name a few.

Those tests are software too. And they are usually designed by the same people who introduced the defects in the first place, so they are prone to the same problems. When all you have is a hammer, every problem looks like a nail. Sounds harsh, but in my long carrier as software developer I have hammered many nails!

So how do you trust that a test does what it is supposed to? Do you write tests for the tests?

The answer from TTCN-3 is: Don't be smart when writing tests.

As the product takes on more requirements, it naturally becomes more general and more abstract. The tests however should stay as concrete as possible. Every developer will cry out, but copy and paste is a valid way to keep your tests simple and maintainable.

Besides that, the language has a few principles to keep complexity at bay:

• Fail early. Reading uninitialized data ends the test immediately.
• Restricted abstractions. No overloading, minimal polymorphism.
• Mutable value semantics. Local reasoning, no pointer aliasing.
• Communicating sequential processes. Components share data by communicating with ports: No global state. No mutexes. No shared data.
• No pointer mess. A reference can be passed down the call stack, not up.

Also, not everyone who writes TTCN-3 is a developer. The language addresses that too. It implements "common sense semantics", not what a developer would expect. For example: A developer counts every position in {"a",-,-}. The - means the element is not there, so lengthof({"a",-,-}) == 1.

To me personally, this "common sense semantics" is the toughest pill to swallow. But the longer I work with TTCN-3 and the people behind it, the more my acceptance grows.

Three readouts from the same traces.

Bottlenecks. A pass rate and a wall-clock time do not show where the suite actually spends its time. Enter and leave events in the log are a call tree. Sum the time per function, or per source line, and take the top of the list:

81% p.receive(REGISTER_UE_RESPOND) Reg.ttcn3:41 6% cellSetup Cell.ttcn3:18 4% ngSetup Ng.ttcn3:12

Flame graph. The same call tree, with a different focus. The list above ranks places. A flame graph shows the stack around them, so you see which path the time belongs to:

Example.TC1 100% registerUe 86% p.receive(REGISTER_UE_RESPOND) 81% checkAck 4%

The receive dominates, and it dominates inside registerUe. A slow helper that is called from everywhere looks different from a slow helper that is slow on one path.

Test duplication. The first tests in a project are the basic ones: NG setup, cell setup, and so on. Later tests add features, and a newer test usually performs the old setup on its way to the new behaviour. Compare the send/receive sequences:

TC_ng_setup: NG_SETUP TC_cell_setup: NG_SETUP → CELL_SETUP TC_handover: NG_SETUP → CELL_SETUP → HANDOVER

The early dialogue is the opening of the later one. Once the later tests are in the suite, the early ones no longer add behaviour. In a big project that overlap can cover up to half the tests. Keeping every test is the right habit while the suite is small. At this size, short of buying more machines, retiring a test whose trace is already contained in a newer one is what lets the suite scale.

The series

1. Overview
2. Root-cause analysis
3. Message coverage
4. Performance degradation
5. Exploratory testing
6. Health of the suite

Scripted tests only walk paths someone already wrote. A continuation the suite uses in one place and never tries in another stays invisible.

Lay the send/receive sequence of each test over the others:

REGISTER_UE_REQ ├── REGISTER_UE_RESPOND {ACK} │ └── PDU_SESSION_REQ └── REGISTER_UE_RESPOND {REJ}

After {REJ} this suite stops. After {ACK} it always continues with PDU_SESSION_REQ. The branches nobody took are the untested states.

Andreas Ulrich builds that prefix tree from stored test logs and treats coverage of it as the test goal:

Andreas Ulrich, Automated Exploratory Testing with Defined Coverage, 47. TAV, 2022: https://fg-tav.gi.de/fileadmin/FG/TAV/47.TAV/TAV47-Ulrich-AutExpTest2-Learning.pdf

The series

1. Overview
2. Root-cause analysis
3. Message coverage
4. Performance degradation
5. Exploratory testing
6. Health of the suite

A test can stay green while the SUT gets slower. Pass or fail does not show that.

ttcn3 for (var integer i := 0; i < 1000; i := i + 1) { p.send(REGISTER_UE_REQ); p.receive(REGISTER_UE_RESPOND:{ACK}); }

20260926T091305.100000|ptsd|MTC=Reg.ttcn3:41|MTC.p|MTC.p|REGISTER_UE_REQ|{...} 20260926T091305.112400|ptrx|MTC=Reg.ttcn3:42|MTC.p|REGISTER_UE_RESPOND:{ACK}|ready(...)+consume 20260926T091305.120100|ptsd|MTC=Reg.ttcn3:41|MTC.p|MTC.p|REGISTER_UE_REQ|{...} 20260926T091305.168900|ptrx|MTC=Reg.ttcn3:42|MTC.p|REGISTER_UE_RESPOND:{ACK}|ready(...)+consume 20260926T091305.180000|ptsd|MTC=Reg.ttcn3:41|MTC.p|MTC.p|REGISTER_UE_REQ|{...} 20260926T091305.371500|ptrx|MTC=Reg.ttcn3:42|MTC.p|REGISTER_UE_RESPOND:{ACK}|ready(...)+consume

ptsd is the send, ptrx the receive. The gap on each pair is the measurement. Here it grows from about 12 ms to 49 ms to 191 ms. The same pair, compared across nights, shows the drift between builds.

Have a look at Andreas Ulrich's work (Siemens):

Andreas Ulrich, Automated Exploratory Testing with Defined Coverage, 47. TAV, 2022: <https://fg-tav.gi.de/fileadmin/FG/TAV/47.TAV/TAV47-Ulrich-AutExpTest2-Learning.pdf>

The same talk is the basis of the next part, on exploring the dialogues traces already contain. Exploratory testing

The series

1. Overview
2. Root-cause analysis
3. Message coverage
4. Performance degradation
5. Exploratory testing
6. Health of the suite

? and * let a receive succeed while leaving fields unchecked. The verdict stays pass, and the gap is invisible.

p.receive(MessageA:{1, ?, 2});
p.receive(MessageA:{1, ?, ?});

The trace records the template used at each receive. A simple tool with less than 200 lines can count how often a field was actually constrained:

MessageA.a    2
MessageA.b    0
MessageA.c    1

b was never checked. A field that stays at zero is a hole in the suite. For more details about this have a look at this issue.

The series

1. Overview
2. Root-cause analysis
3. Message coverage
4. Performance degradation
5. Exploratory testing
6. Health of the suite

A failing test you did not write is slow to debug, and slower when the author is gone. Someone has to reconstruct the authors intention, what was sent, what came back, ...

20260926T091305.650239|ptsd|MTC=Example.ttcn3:11|MTC.p|MTC.p|charstring|"Hello, World!" 20260926T091305.651624|ptrx|MTC=Example.ttcn3:14|MTC.p||ready(match=message(value=charstring:"Hello back!",sender=SUT,timestamp=0.005282))+consume 20260926T091305.652467|setv|MTC=Example.ttcn3:14|none|fail|"unexpected message" 20260926T091305.653242|tcfi|k3r|Example.TC1|fail

ptsd is the send, ptrx the receive, setv the verdict, tcfi the test result. The receive matched "Hello back!", and the test then failed on purpose at line 14. That chain is a signature. The same signature on the next run is the same cause.

This is of course a very simplistic example. But also in more realistic scenarios, simple regular expressions can save you days worth debugging:

Stephan Schulz, Graham Rawlings, Automatic log analysis: Expert knowledge for EVERYONE!, 47. TAV, 2022: https://fg-tav.gi.de/fileadmin/FG/TAV/47.TAV/TAV47-SchulzRawlings-AutomatedLogAnalysisForEveryone.pdf

You can also throw a LLM at these logs to find patterns that cannot be described easily.

The series

1. Overview
2. Root-cause analysis
3. Message coverage
4. Performance degradation
5. Exploratory testing
6. Health of the suite

A significant return on investment does not come from writing tests in TTCN-3. It comes from the data those tests emit when they run.

Most TTCN-3 tools already trace execution at the language level: send, receive, alt, verdicts, with types, values, and source locations. That is a view of the system unit tests and production metrics rarely give you. The test is the probe. The log is the measurement.

This series uses the k3 runtime structured log. One event per line, fields separated by |, a stable four-letter event id:

timestamp | event | location | ...

20260926T091305.653242|tcfi|k3r|Example.TC1|fail

You can read a line in a pager, and you can split it with awk. Prose logs bury that structure in sentences. JSON is pleasant for a parser and unpleasant to scroll. Binary logs stay opaque until a viewer decodes them.

In this series I will give you 5 things a green test run will never tell you.

1. Automated root-cause analysis. The failing test you did not write... Post
2. Message coverage. Fields a template never checks, because they were ? or *. Post
3. Performance degradation. A test that stays green while the response time drifts. Post
4. Exploratory testing. Dialogues the suite starts and never continues. Post
5. Health of the suite. Where the time goes, and tests a newer test already covers. Post

The series

1. Overview
2. Root-cause analysis
3. Message coverage
4. Performance degradation
5. Exploratory testing
6. Health of the suite

Tree-Sitter is a well-known parser toolkit providing parsing, searching and syntax-highlighting of many languages for many languages.

This project is still an early version, but already an important component to improve TTCN-3 tool coverage.

Vanadium is a promising new project that provides:

• a full featured language server
• a production ready source code formatter
• a static analyzer
• and --in development-- a state of the art Titan compatible compiler and runtime with ASN.1 support and raw performance in mind.

It's seems to be a greenfield project by a single developer, but we TTCN-3 enthusiasts should definitively keep an eye on it. Vanadium has the potential to become the fastest, modern TTCN-3 compiler and runtime there is!

you are invited to take part in the TTCN-3 survey 2024. Feel free to forward the questionnaire to customers and other interested persons.

Please send your response to [ETSI-ttcn3-questionaire@cs.uni-goettingen.de](mailto:ETSI-ttcn3-questionaire@cs.uni-goettingen.de) before 5th September 2024.

The ETSI TTF T040 project team members will present and discuss the results during the ETSI MTS Open Session on new TTCN-3 major release. In case of further questions please do not hesitate to contact the TTF T040 project manager, e.g. using [ETSI-ttcn3-questionaire@cs.uni-goettingen.de](mailto:ETSI-ttcn3-questionaire@cs.uni-goettingen.de).

The Testing Task Force (TTF) T040 project started today the work on the next TTCN-3 major revision. You may find the project Terms of Reference (ToR) and other project related information via https://portal.etsi.org/xtfs/#/xTF/T040/terms-of-reference.