A collection of tools for implementing and testing DSLs with Langium!
Explore the docs »
Report Bug
·
Request Feature
This project is a collection of tools, that should power up DSL development with Langium:
Language features:
.toFirstUpper()
Testing features:
vitest
matchers and tools to test DSL validationThis section should list any major frameworks/libraries used to bootstrap your project. Leave any add-ons/plugins for the acknowledgements section. Here are a few examples.
langium-tools
in actionAdd the package to your package.json
npm install langium-tools
You can check out The Umltimative Example: the default langium example project statemachine enhanced with all features of langium-tools
:
When working with Langium documents, it's essential to handle and report issues such as syntax errors, validation errors, and other diagnostics in a unified and user-friendly way. The langium-tools package provides utilities to simplify access to document issues from different sources (lexer, parser, validation) and to generate comprehensive summaries of these issues.
To retrieve issues from a LangiumDocument, use the getDocumentIssues function:
import { getDocumentIssues } from "langium-tools/base";
const issues = getDocumentIssues(document);
By default, getDocumentIssues collects all issues from the lexer, parser, and validation phases. You can customize which issues to include using the optional parameters:
const issues = getDocumentIssues(document, {
skipNonErrorDiagnostics: true, // Skip warnings, information, hints
skipLexerErrors: false, // Include lexer errors
skipParserErrors: false, // Include parser errors
skipValidation: false, // Include validation issues
});
To get a summary of the issues in a document, use the getDocumentIssueSummary function:
import { getDocumentIssueSummary } from "langium-tools/base";
const summary = getDocumentIssueSummary(document);
console.log(summary.summary); // e.g., "1 lexer error(s), 2 validation error(s)"
console.log(summary.message);
/*
Lexer Errors:
Error at 3:15 - Unexpected character
Validation Diagnostics:
Error at 5:10 - Undefined reference to 'myVar'
Warning at 8:5 - Deprecated usage of 'oldFunc'
1 lexer error(s), 2 validation issue(s)
*/
The DocumentIssueSummary object contains:
countTotal
: Total number of issuescountErrors
: Number of issues with severity ERRORcountNonErrors
: Number of issues with severity other than ERRORsummary
: A brief summary stringmessage
: A detailed message listing all issuesYou can convert individual issues to formatted strings using documentIssueToString:
import { documentIssueToString } from "langium-tools/base";
issues.forEach((issue) => {
const issueStr = documentIssueToString(issue);
console.log(issueStr);
});
// Output:
// Error at 3:15 - Unexpected character
// Warning at 5:10 - Deprecated usage of 'oldFunc'
The module defines the following enums for issue sources and severities:
DocumentIssueSource
LEXER
: Issues originating from the lexer (tokenization errors).PARSER
: Issues originating from the parser (syntax errors).VALIDATION
: Issues originating from validation (semantic errors).DocumentIssueSeverity
ERROR
: Critical issues that prevent further processing.WARNING
: Issues that may lead to problems but do not prevent processing.INFORMATION
: Informational messages.HINT
: Suggestions to improve the document.UNKNOWN
: Severity is unknown.For detailed API documentation, see the Typedoc documentation.
The GeneratedContentManager
class provides a unified API for code generation in Langium projects. It allows you to collect generated files in memory, manage multiple output targets, and write the generated content to the filesystem. This is especially useful for snapshot testing, where you can compare the in-memory generated content against saved snapshots to ensure no unintended changes have occurred.
To use the GeneratedContentManager
, you need to:
GeneratedContentManager
, optionally providing a list of workspace URIs.GeneratorManager
.writeToDisk
.import { GeneratedContentManager } from "langium-tools/generator";
// Create a new manager
const manager = new GeneratedContentManager(optionalListOfWorkspaceURIs);
// Generate content for multiple models
generator(manager.generatorManagerFor(model1));
generator(manager.generatorManagerFor(model2));
// Write the generated content to disk
await manager.writeToDisk("./generated");
Here's how you might implement a generator function using GeneratorManager
:
function generator(manager: GeneratorManager) {
const model = manager.getModel();
const document = manager.getDocument();
const workspaceURI = manager.getWorkspaceURI();
const localPath = manager.getDocumentLocalPath();
// Generate files
manager.createFile(
"src-gen/abstract_process.ts",
"// Generated by Langium. Do not edit.",
);
manager.createFile("src/process.ts", "// Initially generated by Langium", {
overwrite: false,
});
}
The GeneratedContentManager
allows you to collect and manage generated files before writing them to disk. This can be particularly useful for testing and validation purposes.
Use the createFile
method of GeneratorManager
to add files to the generated content:
manager.createFile(filePath: string, content: string, options?: CreateFileOptions);
filePath
: The relative path to the file within the output directory.content
: The content to be written to the file.options
: Optional settings, such as overwrite and target.By default, files are overwritten when written to disk. You can control this behavior using the overwrite
option:
manager.createFile("src/process.ts", "// Initially generated by Langium", {
overwrite: false,
});
Targets represent different output directories or configurations. You can define multiple targets with custom settings.
Adding a Target
manager.addTarget({
name: "CUSTOM_TARGET",
overwrite: false,
});
Specify the target when creating files:
manager.createFile("custom.ts", "// Custom target content", {
target: "CUSTOM_TARGET",
});
After generating content, use writeToDisk to write all collected files to the filesystem:
await manager.writeToDisk(outputDir: string, target?: string);
outputDir
: The directory where files will be written.target
: Optional target name. If not provided, the default target is used.For detailed API documentation, see the Typedoc documentation.
import {
GeneratedContentManager,
GeneratorManager,
} from "langium-tools/generator";
// Initialize the manager
const manager = new GeneratedContentManager(["./workspace"]);
// Define a generator function
function generateCode(manager: GeneratorManager) {
const model = manager.getModel();
// Generate code based on the model
manager.createFile("model.ts", `// Code for model ${model.name}`);
}
// Generate code for models
const models = [model1, model2];
models.forEach((model) => {
const genManager = manager.generatorManagerFor(model);
generateCode(genManager);
});
// Write all generated content to disk
await manager.writeToDisk("./generated");
The GeneratedContentManager provides informative error messages when conflicts occur or when file operations fail, helping you quickly identify and resolve issues.
Testing code generators is crucial to ensure that changes in your generator logic do not introduce unintended side effects. The langium-tools
package provides utilities for generator snapshot testing, allowing you to verify that your generators produce the expected output over time.
The main idea is to:
To use generator testing utilities, you need to:
Example Usage
// test/generator.test.ts
import { langiumGeneratorSuite } from "langium-tools/testing";
import { createMyDslServices } from "../src/language/my-dsl-module";
import { generate } from "../src/cli/generator";
describe("Langium code generator tests", () => {
langiumGeneratorSuite("./test-workspaces", {
createServices: () => createMyDslServices,
generateForModel: generate,
});
});
The GeneratorTestOptions
interface allows you to specify how the generator tests should run:
createServices
: Function to create your language services.initWorkspace
: (Optional) Initialize the workspace.buildDocuments
: (Optional) Build documents from the workspace.validateDocuments
: (Optional) Validate the documents before generation.generateForWorkspace
: (Optional) Custom logic to generate content for the entire workspace.generateForModel
: Custom logic to generate content for a single model.In your package.json, you can define scripts to run tests in different modes:
{
"scripts": {
"test": "GENERATOR_TEST=verify vitest run",
"test:generate": "GENERATOR_TEST=generate vitest run"
}
}
The langiumGeneratorSuite
function automatically discovers subdirectories in your test suite directory and runs tests for each DSL workspace.
test-workspaces/
├── workspace1/
│ ├── dsls/
│ │ └── example1.mydsl
│ └── generated/
│ └── generated-code.ts
├── workspace2/
│ ├── dsls/
│ │ └── example2.mydsl
│ └── generated/
│ └── generated-code.ts
npm test
Ensures that the generator output matches the committed generated
directories.
npm run test:generate
Updates the generated
directories with new output from the generators.
When you intentionally change your generator logic, you can:
npm run test:generate
to regenerate the output.git diff
.The generator testing utilities work by:
GeneratedContentManager
.For detailed API documentation, see the Typedoc documentation.
GENERATOR_TEST
environment variable controls the test mode (verify
or generate
).vitest
, but the utilities can be used with any testing framework, if you are not using automatic workspace discovery, but rather define one test for one workspace using langiumGeneratorTest.npm run test:generate
to generate initial snapshots.generated
directories to your version control system.npm test
during development to ensure your generators produce consistent output.npm test
to see if tests fail (they should if output changes).npm run test:generate
to update the snapshots.git diff
.You can customize the behavior by providing custom implementations for optional functions in GeneratorTestOptions
:
initWorkspace
: Customize how the workspace is initialized.buildDocuments
: Control how documents are built from the workspace.validateDocuments
: Implement custom validation logic before generation.generateForWorkspace
: Generate content at the workspace level instead of per model.function customValidateDocuments(
services: DslServices<MyDslService, MyDslSharedService>,
documents: LangiumDocument<MyModel>[],
) {
documents.forEach((doc) => {
const issues = getDocumentIssues(doc);
expect(issues.length).toBe(0);
});
}
langiumGeneratorSuite("./test-workspaces", {
createServices: () => createMyDslServices(),
validateDocuments: customValidateDocuments,
generateForModel: async (services, model, generatorManager) => {
// Generator logic
},
});
The testing utilities provide detailed error messages when:
overwrite
flag.Use Cases
Generator snapshot testing is a powerful technique to maintain the integrity of your code generators. By integrating these testing utilities into your workflow, you can confidently evolve your generators while ensuring consistent output.
The langium-tools
package provides custom Vitest matchers to facilitate testing Langium documents and DSLs. These matchers help you assert that your parsers, validators, and other language services behave as expected. They allow you to check for specific issues, errors, and diagnostics in a clean and expressive way.
expect
function with Langium-specific assertions.To use the Vitest matchers, you need to:
import "langium-tools/testing"; // Ensure matchers are registered
import { parseHelper } from "langium/test";
You can use the parseWithMarks
function to parse your DSL code and extract markers. Markers are special annotations in your code that help you specify exact locations for expected issues.
import { parseWithMarks } from "langium-tools/testing";
import { adjusted } from "langium-tools/base";
const services = createMyDslServices(EmptyFileSystem);
const doParse = parseHelper<Model>(services.MyStateMachine);
const parse = (input: string) => doParse(input, { validate: true });
// Parse the code with markers
const parsedDocument = await parseWithMarks(
parse,
adjusted`
statemachine MyStatemachine
state [[[UnusedRule]]];
`,
"[[[",
"]]]",
);
toHaveNoErrors
Asserts that a Langium document has no errors after parsing. By default, non-error diagnostics (warnings, hints) are ignored.
expect(document).toHaveNoErrors();
parameters
(optional): Customize which types of issues to ignore.test("document has no errors", async () => {
const document = await parse("validCode");
expect(document).toHaveNoErrors();
});
toHaveNoIssues
Asserts that a Langium document has no issues (errors, warnings, hints) after parsing.
expect(document).toHaveNoIssues();
parameters
(optional): Customize which types of issues to ignore.test("document has no issues", async () => {
const document = await parse("validCode");
expect(document).toHaveNoIssues();
});
toHaveDocumentIssues
Asserts that a parsed document with markers has specific issues at specified markers.
expect(parsedDocument).toHaveDocumentIssues(expectedIssues, parameters?);
expectedIssues
: An array of issue expectations.parameters
(optional): Customize which types of issues to ignore.interface IssueExpectation {
severity?: DocumentIssueSeverity; // Defaults to ERROR
source?: DocumentIssueSource; // LEXER, PARSER, VALIDATION
message: string | RegExp; // Expected message
markerId?: number; // 0-based index of the marker
}
test("document has expected issues", async () => {
const dslCode = adjusted`
grammar MyLanguage
entry Rule: name=ID;
[[[UnusedRule]]]: name=ID;
terminal ID: /\\^?[_a-zA-Z][\\w_]*/;
`;
const parsedDocument = await parseWithMarks(parse, dslCode, "[[[", "]]]");
expect(parsedDocument).toHaveDocumentIssues([
{
source: DocumentIssueSource.VALIDATION,
severity: DocumentIssueSeverity.WARNING,
message: "This rule is declared but never referenced.",
markerId: 0,
},
]);
});
toContainIssue
Asserts that a Langium document contains a specific issue.
expect(document).toContainIssue(expectedIssue, parameters?);
expectedIssue
: The issue expectation.parameters
(optional): Customize which types of issues to ignore.test("document contains specific issue", async () => {
const document = await parse("invalidCode");
expect(document).toContainIssue({
message: /Unexpected character/,
severity: DocumentIssueSeverity.ERROR,
source: DocumentIssueSource.LEXER,
});
});
test("document has no errors", async () => {
const document = await parse("validCode");
expect(document).toHaveNoErrors();
});
test("document has expected validation issue at marker", async () => {
const dslCode = adjusted`
grammar MyLanguage
entry Rule: name=ID;
<<<UnusedRule>>>: name=ID;
terminal ID: /\\^?[_a-zA-Z][\\w_]*/;
`;
const parsedDocument = await parseWithMarks(parse, dslCode, "<<<", ">>>");
expect(parsedDocument).toHaveDocumentIssues([
{
source: DocumentIssueSource.VALIDATION, // Optional
severity: DocumentIssueSeverity.WARNING, // Optional
message: "This rule is declared but never referenced.",
markerId: 0, // Optional
},
]);
});
test("document contains issue matching regex", async () => {
const dslCode = adjusted`
grammar MyLanguage
entry Rule: <<<name>>> ID;
terminal ID: /\\^?[_a-zA-Z][\\w_]*/;
`;
const parsedDocument = await parseWithMarks(parse, dslCode, "<<<", ">>>");
expect(parsedDocument).toContainIssue({
message: /Could not resolve reference to AbstractRule named '.\*'/,
markerId: 0,
});
});
All matchers accept an optional parameters object to customize which issues to include or ignore in your assertions.
interface IgnoreParameters {
ignoreParserErrors?: boolean; // Default: false
ignoreLexerErrors?: boolean; // Default: false
ignoreValidationErrors?: boolean; // Default: false
ignoreNonErrorDiagnostics?: boolean; // Default: depends on matcher
}
test("document has no errors, ignoring lexer errors", async () => {
const document = await parse("codeWithLexerErrors");
expect(document).toHaveNoErrors({ ignoreLexerErrors: true });
});
langium-tools/testing
.adjusted
Function for TemplatesIn the examples, you might notice the use of a adjusted
function to format template strings. This is a utility to clean up multiline strings in tests.
See adjusted
typedoc for more information.
The Vitest matchers provided by langium-tools
enhance your testing capabilities by allowing you to write expressive and precise assertions for your Langium documents. By utilizing markers and custom matchers, you can create robust tests that ensure your language services work as intended.
For more detailed information, refer to the API documentation.
adjusted
The adjusted function is a utility for formatting multi-line template strings, ensuring that indentation and line breaks are preserved correctly. This is particularly useful for generating code snippets or structured text where indentation is essential.
See adjusted
typedoc for more information.
adjustedUnix
for Unix-style newlines (\n
)The adjusted function is used as a tagged template literal:
import { adjusted } from "langium-tools/base";
const text = adjusted`
${cmds}
if (sayAgain) {
${cmds}
}
`;
console.log(text);
Output:
console.log('Hello,');
console.log('World!');
if (sayAgain) {
console.log('Hello,');
console.log('World!');
}
Notice, how indentation of the dynamic placeholders is adjusted to match the surrounding code.
The function returns a formatted string with the following properties:
JavaImportManager
and generateJavaFile
The JavaImportManager
and generateJavaFile
utilities simplify the process of generating well-structured Java code within your Langium-based projects. They handle common boilerplate code such as package declarations and import statements, allowing you to focus on the core content of your Java classes.
To use these utilities, you need to:
import { expandToNode } from "langium/generate";
import { generateJavaFile } from "langium-tools/lang/java";
import { GeneratorManager } from "langium-tools/generator";
generateJavaFile
The generateJavaFile
function generates a Java source file with proper package declaration, imports, and body content. It leverages JavaImportManager
to manage imports automatically based on the classes used in the body content.
generateJavaFile(
fileName: string,
packageName: string,
generatorManager: GeneratorManager,
bodyGenerator: (importManager: (fqn: string) => string) => CompositeGeneratorNode,
options?: CreateFileOptions
): void;
.java
extension).com.example.project
).generateJavaFile("MyClass", "com.example.project", generatorManager, (imp) => expandToNode`
public class MyClass {
public ${imp("java.util.Properties")} getProperties() {
// ...
}
}
`),
);
Imports:
imp
function is used to import java.util.Properties
. It returns the appropriate class name (Properties
in this case), handling imports automatically.Generated Output:
package com.example.project;
import java.util.Properties;
public class MyClass {
public Properties getProperties() {
// ...
}
}
The JavaImportManager
ensures that:
java.lang
) are not imported unnecessarily. private ${imp('com.example.package1.MyClass')} myClass1;
private ${imp('com.example.package2.MyClass')} myClass2;
);
Generated imports:
import com.example.package1.MyClass;
Generated output:
private MyClass myClass1;
private com.example.package2.MyClass myClass2;
The JavaImportManager
and generateJavaFile
utilities provide powerful tools for generating Java code in a structured and efficient manner. They handle the complexities of import management and file generation, allowing you to focus on implementing the logic of your code generation.
By integrating these utilities into your Langium projects, you can automate Java code generation with ease and confidence, ensuring that your generated code is clean, consistent, and maintainable.
For more detailed information, refer to the API documentation linked above.
langium-tools
and langium-test-tools
NPM packages to prevent test helper classes to be released with production codeSee the open issues for a full list of proposed features (and known issues).
Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.
If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!
Fork the Project
Create your Feature Branch (git checkout -b feature/AmazingFeature
)
Commit your Changes (git commit -m 'Add some AmazingFeature'
)
Open a Pull Request
Release with release-it
Distributed under the MIT License. See LICENSE.txt
for more information.
Boris Brodski - @BorisBrodski - brodsky_boris@yahoo.com
Project Link: https://github.com/borisbrodski/langium-tools