langium-tools - v0.1.2

Contributors Forks Stargazers Issues MIT License


langium-tools

A collection of tools for implementing and testing DSLs with Langium!

Explore the docs »

Report Bug · Request Feature

  1. About The Project
  2. Getting Started
  3. Documentation
  4. Roadmap
  5. Contributing
  6. License
  7. Contact
  8. Acknowledgments

This project is a collection of tools, that should power up DSL development with Langium:

Language features:

  • Helper methods - collection of helper methods, like .toFirstUpper()
  • Generators - nice framework to generate files from AST (both in memory and on the disk)
  • Issues - unified way to access, verify and print out errors and warning in DSLs

Testing features:

  • Advance generator testing framework (optimized for large amount of generated code)
  • vitest matchers and tools to test DSL validation

(back to top)

This section should list any major frameworks/libraries used to bootstrap your project. Leave any add-ons/plugins for the acknowledgements section. Here are a few examples.

  • Typescript

(back to top)

  • Add package to your project
  • Start using one or more features it provides
  • Check out example project to see langium-tools in action

Add the package to your package.json

  1. Install NPM packages
    npm install langium-tools
    

    (back to top)

You can check out The Umltimative Example: the default langium example project statemachine enhanced with all features of langium-tools:

  • TODO - ADD LINK HERE

See typedoc documentation

When working with Langium documents, it's essential to handle and report issues such as syntax errors, validation errors, and other diagnostics in a unified and user-friendly way. The langium-tools package provides utilities to simplify access to document issues from different sources (lexer, parser, validation) and to generate comprehensive summaries of these issues.

  • Unified Issue Access: Collect all issues from a LangiumDocument, including lexer errors, parser errors, and validation diagnostics.
  • Customizable Filtering: Optionally filter issues based on their source or severity.
  • Issue Summarization: Generate summaries of issues with counts and formatted messages.
  • Formatted Issue Strings: Convert issues into human-readable strings for logging or displaying.

To retrieve issues from a LangiumDocument, use the getDocumentIssues function:

import { getDocumentIssues } from "langium-tools/base";

const issues = getDocumentIssues(document);

By default, getDocumentIssues collects all issues from the lexer, parser, and validation phases. You can customize which issues to include using the optional parameters:

const issues = getDocumentIssues(document, {
skipNonErrorDiagnostics: true, // Skip warnings, information, hints
skipLexerErrors: false, // Include lexer errors
skipParserErrors: false, // Include parser errors
skipValidation: false, // Include validation issues
});

To get a summary of the issues in a document, use the getDocumentIssueSummary function:

import { getDocumentIssueSummary } from "langium-tools/base";

const summary = getDocumentIssueSummary(document);

console.log(summary.summary); // e.g., "1 lexer error(s), 2 validation error(s)"
console.log(summary.message);
/*
Lexer Errors:
Error at 3:15 - Unexpected character

Validation Diagnostics:
Error at 5:10 - Undefined reference to 'myVar'
Warning at 8:5 - Deprecated usage of 'oldFunc'

1 lexer error(s), 2 validation issue(s)
*/

The DocumentIssueSummary object contains:

  • countTotal: Total number of issues
  • countErrors: Number of issues with severity ERROR
  • countNonErrors: Number of issues with severity other than ERROR
  • summary: A brief summary string
  • message: A detailed message listing all issues

You can convert individual issues to formatted strings using documentIssueToString:

import { documentIssueToString } from "langium-tools/base";

issues.forEach((issue) => {
const issueStr = documentIssueToString(issue);
console.log(issueStr);
});

// Output:
// Error at 3:15 - Unexpected character
// Warning at 5:10 - Deprecated usage of 'oldFunc'

The module defines the following enums for issue sources and severities:

DocumentIssueSource

  • LEXER: Issues originating from the lexer (tokenization errors).
  • PARSER: Issues originating from the parser (syntax errors).
  • VALIDATION: Issues originating from validation (semantic errors).

DocumentIssueSeverity

  • ERROR: Critical issues that prevent further processing.
  • WARNING: Issues that may lead to problems but do not prevent processing.
  • INFORMATION: Informational messages.
  • HINT: Suggestions to improve the document.
  • UNKNOWN: Severity is unknown.

For detailed API documentation, see the Typedoc documentation.

  • The position information in issues is zero-based for both lines and columns.
  • The utilities handle the potential absence of position data gracefully.
  • Ensure that you have imported the necessary functions from langium-tools in your project.

(back to top)

The GeneratedContentManager class provides a unified API for code generation in Langium projects. It allows you to collect generated files in memory, manage multiple output targets, and write the generated content to the filesystem. This is especially useful for snapshot testing, where you can compare the in-memory generated content against saved snapshots to ensure no unintended changes have occurred.

  • Unified Generator API: Simplify code generation with a consistent interface.
  • In-Memory Generation: Collect generated files in memory for flexible processing.
  • Initial Generation: Don't overwrite existing files.
  • Preserve File Timestamp: File will be overwritten only if the content has actually changed.
  • Multiple Targets: Support multiple output directories (targets) with custom overwrite settings.
  • Snapshot Testing Support: Facilitate testing by comparing in-memory generated content with committed snapshots.

To use the GeneratedContentManager, you need to:

  1. Create an instance of GeneratedContentManager, optionally providing a list of workspace URIs.
  2. Generate content for your models using GeneratorManager.
  3. Write the generated content to disk using writeToDisk.
import { GeneratedContentManager } from "langium-tools/generator";

// Create a new manager
const manager = new GeneratedContentManager(optionalListOfWorkspaceURIs);

// Generate content for multiple models
generator(manager.generatorManagerFor(model1));
generator(manager.generatorManagerFor(model2));

// Write the generated content to disk
await manager.writeToDisk("./generated");

Here's how you might implement a generator function using GeneratorManager:

function generator(manager: GeneratorManager) {
const model = manager.getModel();
const document = manager.getDocument();
const workspaceURI = manager.getWorkspaceURI();
const localPath = manager.getDocumentLocalPath();

// Generate files
manager.createFile(
"src-gen/abstract_process.ts",
"// Generated by Langium. Do not edit.",
);
manager.createFile("src/process.ts", "// Initially generated by Langium", {
overwrite: false,
});
}

The GeneratedContentManager allows you to collect and manage generated files before writing them to disk. This can be particularly useful for testing and validation purposes.

Use the createFile method of GeneratorManager to add files to the generated content:

manager.createFile(filePath: string, content: string, options?: CreateFileOptions);
  • filePath: The relative path to the file within the output directory.
  • content: The content to be written to the file.
  • options: Optional settings, such as overwrite and target.

By default, files are overwritten when written to disk. You can control this behavior using the overwrite option:

manager.createFile("src/process.ts", "// Initially generated by Langium", {
overwrite: false,
});

Targets represent different output directories or configurations. You can define multiple targets with custom settings.

Adding a Target

manager.addTarget({
name: "CUSTOM_TARGET",
overwrite: false,
});

Specify the target when creating files:

manager.createFile("custom.ts", "// Custom target content", {
target: "CUSTOM_TARGET",
});

After generating content, use writeToDisk to write all collected files to the filesystem:

await manager.writeToDisk(outputDir: string, target?: string);
  • outputDir: The directory where files will be written.
  • target: Optional target name. If not provided, the default target is used.

For detailed API documentation, see the Typedoc documentation.

  • Workspace URIs: When creating a GeneratedContentManager, you can provide workspace URIs to help determine relative paths for documents.
  • Conflict Detection: The manager detects conflicts if the same file is generated multiple times with different content or overwrite settings.
  • Asynchronous Operations: Writing to disk is asynchronous and returns a promise.
import {
GeneratedContentManager,
GeneratorManager,
} from "langium-tools/generator";

// Initialize the manager
const manager = new GeneratedContentManager(["./workspace"]);

// Define a generator function
function generateCode(manager: GeneratorManager) {
const model = manager.getModel();
// Generate code based on the model
manager.createFile("model.ts", `// Code for model ${model.name}`);
}

// Generate code for models
const models = [model1, model2];
models.forEach((model) => {
const genManager = manager.generatorManagerFor(model);
generateCode(genManager);
});

// Write all generated content to disk
await manager.writeToDisk("./generated");

The GeneratedContentManager provides informative error messages when conflicts occur or when file operations fail, helping you quickly identify and resolve issues.

  • Modular Code Generation: Manage code generation for multiple models or modules in a unified way.
  • Testing Generators: Use in-memory content collection to test your generators without writing to disk.
  • Custom Build Pipelines: Integrate with build systems by controlling when and how generated content is written.

(back to top)

Testing code generators is crucial to ensure that changes in your generator logic do not introduce unintended side effects. The langium-tools package provides utilities for generator snapshot testing, allowing you to verify that your generators produce the expected output over time.

The main idea is to:

  • Commit generated content to your version control system (e.g., Git).
  • Run tests in verify mode to ensure generators have not changed unintentionally.
  • Approve changes by regenerating snapshots and reviewing diffs when intentional changes are made.
  • Automated Generator Tests: Run tests across multiple DSL workspaces to verify generator output.
  • Snapshot Verification: Compare in-memory generated content against committed snapshots.
  • Flexible Modes: Switch between generate mode (to update snapshots) and verify mode (to validate output).

To use generator testing utilities, you need to:

  1. Define generator test options.
  2. Set up test suites using langiumGeneratorSuite.
  3. Configure your test scripts to handle generate and verify modes.

Example Usage

// test/generator.test.ts
import { langiumGeneratorSuite } from "langium-tools/testing";
import { createMyDslServices } from "../src/language/my-dsl-module";
import { generate } from "../src/cli/generator";

describe("Langium code generator tests", () => {
langiumGeneratorSuite("./test-workspaces", {
createServices: () => createMyDslServices,
generateForModel: generate,
});
});

The GeneratorTestOptions interface allows you to specify how the generator tests should run:

  • createServices: Function to create your language services.
  • initWorkspace: (Optional) Initialize the workspace.
  • buildDocuments: (Optional) Build documents from the workspace.
  • validateDocuments: (Optional) Validate the documents before generation.
  • generateForWorkspace: (Optional) Custom logic to generate content for the entire workspace.
  • generateForModel: Custom logic to generate content for a single model.

In your package.json, you can define scripts to run tests in different modes:

{
"scripts": {
"test": "GENERATOR_TEST=verify vitest run",
"test:generate": "GENERATOR_TEST=generate vitest run"
}
}
  • Verify Mode: Runs tests to verify that the generator output matches the committed snapshots.
  • Generate Mode: Regenerates the snapshots and updates the committed files.

The langiumGeneratorSuite function automatically discovers subdirectories in your test suite directory and runs tests for each DSL workspace.

test-workspaces/
├── workspace1/
│ ├── dsls/
│ │ └── example1.mydsl
│ └── generated/
│ └── generated-code.ts
├── workspace2/
│ ├── dsls/
│ │ └── example2.mydsl
│ └── generated/
│ └── generated-code.ts
  • Verify Mode:
npm test

Ensures that the generator output matches the committed generated directories.

  • Generate Mode:
npm run test:generate

Updates the generated directories with new output from the generators.

When you intentionally change your generator logic, you can:

  1. Run npm run test:generate to regenerate the output.
  2. Review the changes using git diff.
  3. Commit the updated generated content to approve the changes.

The generator testing utilities work by:

  • Collecting generated content in memory using GeneratedContentManager.
  • Writing the generated content to disk in generate mode.
  • Comparing the in-memory content against the committed files in verify mode.

For detailed API documentation, see the Typedoc documentation.

  • Environment Variable: The GENERATOR_TEST environment variable controls the test mode (verify or generate).
  • Testing Framework: The examples use vitest, but the utilities can be used with any testing framework, if you are not using automatic workspace discovery, but rather define one test for one workspace using langiumGeneratorTest.
  1. Initial Setup:
  • Write your generator logic.
  • Run npm run test:generate to generate initial snapshots.
  • Commit the generated directories to your version control system.
  1. Continuous Testing:
  • Run npm test during development to ensure your generators produce consistent output.
  1. Updating Generators:
  • Make changes to your generator logic.
  • Run npm test to see if tests fail (they should if output changes).
  • Run npm run test:generate to update the snapshots.
  • Review changes with git diff.
  • Commit the changes to approve them.

You can customize the behavior by providing custom implementations for optional functions in GeneratorTestOptions:

  • initWorkspace: Customize how the workspace is initialized.
  • buildDocuments: Control how documents are built from the workspace.
  • validateDocuments: Implement custom validation logic before generation.
  • generateForWorkspace: Generate content at the workspace level instead of per model.
function customValidateDocuments(
services: DslServices<MyDslService, MyDslSharedService>,
documents: LangiumDocument<MyModel>[],
) {
documents.forEach((doc) => {
const issues = getDocumentIssues(doc);
expect(issues.length).toBe(0);
});
}

langiumGeneratorSuite("./test-workspaces", {
createServices: () => createMyDslServices(),
validateDocuments: customValidateDocuments,
generateForModel: async (services, model, generatorManager) => {
// Generator logic
},
});

The testing utilities provide detailed error messages when:

  • Generated files do not match the committed snapshots.
  • Unexpected files are found in the generated directories.
  • Metadata mismatches occur, e.g. overwrite flag.

Use Cases

  • Regression Testing: Ensure that changes to your generators do not introduce regressions.
  • Collaboration: Facilitate code reviews by providing clear diffs of generator output.
  • Automation: Integrate into CI/CD pipelines for automated testing.

Generator snapshot testing is a powerful technique to maintain the integrity of your code generators. By integrating these testing utilities into your workflow, you can confidently evolve your generators while ensuring consistent output.

(back to top)

The langium-tools package provides custom Vitest matchers to facilitate testing Langium documents and DSLs. These matchers help you assert that your parsers, validators, and other language services behave as expected. They allow you to check for specific issues, errors, and diagnostics in a clean and expressive way.

  • Custom Matchers: Extend Vitest's expect function with Langium-specific assertions.
  • Issue Assertions: Assert the presence or absence of specific issues in a document.
  • Marker-Based Testing: Use markers in your DSL code to pinpoint locations for expected issues.
  • Flexible Ignoring: Customize which types of issues (lexer, parser, validation) to include or ignore in your assertions.

To use the Vitest matchers, you need to:

  1. Import the matchers in your test files.
  2. Parse your DSL code, optionally using markers.
  3. Use the custom matchers in your assertions.
import "langium-tools/testing"; // Ensure matchers are registered
import { parseHelper } from "langium/test";

You can use the parseWithMarks function to parse your DSL code and extract markers. Markers are special annotations in your code that help you specify exact locations for expected issues.

import { parseWithMarks } from "langium-tools/testing";
import { adjusted } from "langium-tools/base";

const services = createMyDslServices(EmptyFileSystem);
const doParse = parseHelper<Model>(services.MyStateMachine);
const parse = (input: string) => doParse(input, { validate: true });

// Parse the code with markers
const parsedDocument = await parseWithMarks(
parse,
adjusted`
statemachine MyStatemachine

state [[[UnusedRule]]];
`,
"[[[",
"]]]",
);

toHaveNoErrors

Asserts that a Langium document has no errors after parsing. By default, non-error diagnostics (warnings, hints) are ignored.

expect(document).toHaveNoErrors();
  • parameters (optional): Customize which types of issues to ignore.
test("document has no errors", async () => {
const document = await parse("validCode");
expect(document).toHaveNoErrors();
});

toHaveNoIssues

Asserts that a Langium document has no issues (errors, warnings, hints) after parsing.

expect(document).toHaveNoIssues();
  • parameters (optional): Customize which types of issues to ignore.
test("document has no issues", async () => {
const document = await parse("validCode");
expect(document).toHaveNoIssues();
});

toHaveDocumentIssues

Asserts that a parsed document with markers has specific issues at specified markers.

expect(parsedDocument).toHaveDocumentIssues(expectedIssues, parameters?);
  • expectedIssues: An array of issue expectations.
  • parameters (optional): Customize which types of issues to ignore.
interface IssueExpectation {
severity?: DocumentIssueSeverity; // Defaults to ERROR
source?: DocumentIssueSource; // LEXER, PARSER, VALIDATION
message: string | RegExp; // Expected message
markerId?: number; // 0-based index of the marker
}
test("document has expected issues", async () => {
const dslCode = adjusted`
grammar MyLanguage
entry Rule: name=ID;
[[[UnusedRule]]]: name=ID;
terminal ID: /\\^?[_a-zA-Z][\\w_]*/;
`;
const parsedDocument = await parseWithMarks(parse, dslCode, "[[[", "]]]");

expect(parsedDocument).toHaveDocumentIssues([
{
source: DocumentIssueSource.VALIDATION,
severity: DocumentIssueSeverity.WARNING,
message: "This rule is declared but never referenced.",
markerId: 0,
},
]);
});

toContainIssue

Asserts that a Langium document contains a specific issue.

expect(document).toContainIssue(expectedIssue, parameters?);
  • expectedIssue: The issue expectation.
  • parameters (optional): Customize which types of issues to ignore.
test("document contains specific issue", async () => {
const document = await parse("invalidCode");
expect(document).toContainIssue({
message: /Unexpected character/,
severity: DocumentIssueSeverity.ERROR,
source: DocumentIssueSource.LEXER,
});
});
test("document has no errors", async () => {
const document = await parse("validCode");
expect(document).toHaveNoErrors();
});
test("document has expected validation issue at marker", async () => {
const dslCode = adjusted`
grammar MyLanguage
entry Rule: name=ID;
<<<UnusedRule>>>: name=ID;
terminal ID: /\\^?[_a-zA-Z][\\w_]*/;
`;
const parsedDocument = await parseWithMarks(parse, dslCode, "<<<", ">>>");

expect(parsedDocument).toHaveDocumentIssues([
{
source: DocumentIssueSource.VALIDATION, // Optional
severity: DocumentIssueSeverity.WARNING, // Optional
message: "This rule is declared but never referenced.",
markerId: 0, // Optional
},
]);
});
test("document contains issue matching regex", async () => {
const dslCode = adjusted`
grammar MyLanguage
entry Rule: <<<name>>> ID;
terminal ID: /\\^?[_a-zA-Z][\\w_]*/;
`;
const parsedDocument = await parseWithMarks(parse, dslCode, "<<<", ">>>");

expect(parsedDocument).toContainIssue({
message: /Could not resolve reference to AbstractRule named '.\*'/,
markerId: 0,
});
});

All matchers accept an optional parameters object to customize which issues to include or ignore in your assertions.

interface IgnoreParameters {
ignoreParserErrors?: boolean; // Default: false
ignoreLexerErrors?: boolean; // Default: false
ignoreValidationErrors?: boolean; // Default: false
ignoreNonErrorDiagnostics?: boolean; // Default: depends on matcher
}
test("document has no errors, ignoring lexer errors", async () => {
const document = await parse("codeWithLexerErrors");
expect(document).toHaveNoErrors({ ignoreLexerErrors: true });
});
  • Extending Vitest: The matchers are added to Vitest's expect function by importing langium-tools/testing.
  • Error Messages: The matchers provide detailed error messages to help you diagnose failing tests.

In the examples, you might notice the use of a adjusted function to format template strings. This is a utility to clean up multiline strings in tests.

See adjusted typedoc for more information.

The Vitest matchers provided by langium-tools enhance your testing capabilities by allowing you to write expressive and precise assertions for your Langium documents. By utilizing markers and custom matchers, you can create robust tests that ensure your language services work as intended.

For more detailed information, refer to the API documentation.

(back to top)

The adjusted function is a utility for formatting multi-line template strings, ensuring that indentation and line breaks are preserved correctly. This is particularly useful for generating code snippets or structured text where indentation is essential.

See adjusted typedoc for more information.

  • Automatic Indentation: Dynamically inserted placeholders (${...}) are adjusted to match the indentation level of the placeholder itself, making it easy to maintain proper structure in the output.
  • New lines: Platform-style newlines, see adjustedUnix for Unix-style newlines (\n)

The adjusted function is used as a tagged template literal:

import { adjusted } from "langium-tools/base";
const text = adjusted`
${cmds}
if (sayAgain) {
${cmds}
}
`;

console.log(text);

Output:

 console.log('Hello,');
console.log('World!');
if (sayAgain) {
console.log('Hello,');
console.log('World!');
}

Notice, how indentation of the dynamic placeholders is adjusted to match the surrounding code.

  • staticParts: The static (literal) parts of the template string.
  • substitutions: The dynamic parts of the template string, which will be injected into the corresponding placeholders in staticParts.

The function returns a formatted string with the following properties:

  • Indentation of dynamic placeholders is adjusted to match the indentation of the placeholder in the template.

(back to top)

The JavaImportManager and generateJavaFile utilities simplify the process of generating well-structured Java code within your Langium-based projects. They handle common boilerplate code such as package declarations and import statements, allowing you to focus on the core content of your Java classes.

  • JavaImportManager: Manages Java class imports during code generation, automatically handling import statements and resolving class name collisions.
  • generateJavaFile: A utility function that streamlines the creation of Java source files with proper package declarations, imports, and class content.

To use these utilities, you need to:

  • Import the necessary modules in your code.
  • Define your class content using CompositeGeneratorNode and the provided importManager.
  • Generate the Java file using generateJavaFile.
import { expandToNode } from "langium/generate";
import { generateJavaFile } from "langium-tools/lang/java";
import { GeneratorManager } from "langium-tools/generator";

The generateJavaFile function generates a Java source file with proper package declaration, imports, and body content. It leverages JavaImportManager to manage imports automatically based on the classes used in the body content.

generateJavaFile(
fileName: string,
packageName: string,
generatorManager: GeneratorManager,
bodyGenerator: (importManager: (fqn: string) => string) => CompositeGeneratorNode,
options?: CreateFileOptions
): void;
  • fileName: Name of the Java file to be generated (without the .java extension).
  • packageName: The package name for the Java file (e.g., com.example.project).
  • generatorManager: Manages code generation for a specific model.
  • bodyGenerator: A function that generates the body of the Java file, receiving an importManager function to handle class imports.
  • options (optional): Settings for file creation, such as encoding or overwrite behavior and target directory.
generateJavaFile("MyClass", "com.example.project", generatorManager, (imp) => expandToNode`
public class MyClass {
public ${imp("java.util.Properties")} getProperties() {
// ...
}
}
`),
);
  • Imports:

    • The passed imp function is used to import java.util.Properties. It returns the appropriate class name (Properties in this case), handling imports automatically.
  • Generated Output:

package com.example.project;

import java.util.Properties;

public class MyClass {
public Properties getProperties() {
// ...
}
}

The JavaImportManager ensures that:

  • Classes from the same package or default packages (like java.lang) are not imported unnecessarily.
  • Class name collisions are resolved by using fully qualified names when needed.
  • Imports are sorted alphabetically, and duplicates are avoided.
    private ${imp('com.example.package1.MyClass')} myClass1;
private ${imp('com.example.package2.MyClass')} myClass2;
);

Generated imports:

import com.example.package1.MyClass;

Generated output:

    private MyClass myClass1;
private com.example.package2.MyClass myClass2;
  • Classes:
    • JavaImportManager
  • Functions:
    • generateJavaFile
    • adjusted
  • Interfaces:
    • GeneratorManager
    • CreateFileOptions

The JavaImportManager and generateJavaFile utilities provide powerful tools for generating Java code in a structured and efficient manner. They handle the complexities of import management and file generation, allowing you to focus on implementing the logic of your code generation.

By integrating these utilities into your Langium projects, you can automate Java code generation with ease and confidence, ensuring that your generated code is clean, consistent, and maintainable.

For more detailed information, refer to the API documentation linked above.

(back to top)

  • [ ] Split packages into langium-tools and langium-test-tools NPM packages to prevent test helper classes to be released with production code

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

  1. Fork the Project

  2. Create your Feature Branch (git checkout -b feature/AmazingFeature)

  3. Commit your Changes (git commit -m 'Add some AmazingFeature')

  4. Open a Pull Request

  5. Release with release-it

contrib.rocks image

(back to top)

Distributed under the MIT License. See LICENSE.txt for more information.

(back to top)

Boris Brodski - @BorisBrodski - brodsky_boris@yahoo.com

Project Link: https://github.com/borisbrodski/langium-tools

(back to top)

(back to top)