diff --git a/docs/example/runmacro_example.zip b/docs/example/runmacro_example.zip index 44b5f62..9b53541 100644 Binary files a/docs/example/runmacro_example.zip and b/docs/example/runmacro_example.zip differ diff --git a/docs/source/diffpy-macro.rst b/docs/source/diffpy-macro.rst index cd006bc..587eaca 100644 --- a/docs/source/diffpy-macro.rst +++ b/docs/source/diffpy-macro.rst @@ -7,7 +7,7 @@ Diffpy macro files can be executed using the ``runmacro`` application. .. code-block:: bash - diffpy-app runmacro + diffpy-app runmacro To follow the example, @@ -30,105 +30,89 @@ To follow the example, How to write macro ~~~~~~~~~~~~~~~~~~ +A macro file is a list of calls to the methods of the refinement session, +one call per line or spread over several lines. Arguments are written as +``name=value``. Supported values are numbers, quoted strings, ``True``, +``False``, ``None``, lists ``[...]``, and dictionaries ``{"key": value}``. +Everything after ``#`` is a comment. Calls are executed in order, and the +macro is rejected before running if any method name is unknown. + Let's still use the Ni example created earlier, but this time we will write the macro file from scratch. -1. Prepare the structure and profile file you want to work with. - -.. note:: - - The structure file is a ``.cif`` file representing the atomic arrangement in - your sample, and the profile file is a ``.gr`` file containing the - experimental data. - -Start the macro file with the following two lines: +1. Load the profile and set the calculation range. .. code-block:: text - load structure G1 from "path/to/structure.cif" - load profile exp_ni from "path/to/profile.gr" + add_profile_from_file( + profile_name="ni_profile", profile_path="path/to/profile.gr" + ) + set_profile_calculation_range( + profile_name="ni_profile", xmin=1.5, xmax=20, dx=0.01 + ) + update_profile_meta(profile_name="ni_profile", meta={"qmin": 0.1}) -``G1`` and ``exp_ni`` are the identifiers used in the macro to refer to the -structure and profile files, respectively. Quotes ``""`` are must to specify -the file paths. +``ni_profile`` is the identifier used in the macro to refer to the profile. +The profile file is a ``.gr`` file containing the experimental data. +``xmin``, ``xmax`` and ``dx`` set the range and step size of the calculation, +and ``qmin`` in the profile meta sets the minimum Q used in the refinement. -2. (Optional) Declare the space group of the structure. +2. Create the PDF model from the structure file. .. code-block:: text - set G1 spacegroup as Fm-3m + add_pdf_model( + model_name="pdf", structure_file_path="path/to/structure.cif", finite=False + ) + constrain_pdf_model_space_group_symmetry( + model_name="pdf", space_group="Fm-3m" + ) -``G1`` is the identifier for the structure file loaded earlier. Space group -symmetry ``Fm-3m`` is preserved during the refinement. You can also set -it to be ``auto`` to use the one automatically parsed from the structure file. -But if space group is not provided, nor can it be determined from the structure -file, it will be considered as ``P1`` space group. +The structure file is a ``.cif`` file representing the atomic arrangement in +your sample. The space group symmetry ``Fm-3m`` is preserved during the +refinement. -3. (Optional) Set the calculation parameters for the refinement. +3. Set the initial values of the variables. .. code-block:: text - set exp_ni calculation_range as 0.5 20.0 0.01 # r_min, r_max, r_step - set exp_ni q_range as 0.5 20.0 # q_min, q_max + set_variables_value( + name_value_dict={ + "pdf.phase.lattice.a": 3.52, + "pdf.scale": 0.4, + "pdf.phase.Ni0.Uiso": 0.005, + "pdf.delta2": 2, + "pdf.qdamp": 0.04, + "pdf.qbroad": 0.02, + } + ) -``exp_ni`` is the identifier for the profile file loaded earlier. -``calculation_range`` specifies the range and step size for the calculation, -while ``q_range`` specifies the range of Q values to be used in the refinement. -If calculation parameters are not set, it will use the ones -that are defined in the profile file. +Variable names follow the format ``model_name.parameter``. Parameters of the +structure are under ``model_name.phase``, for example +``pdf.phase.Ni0.Uiso`` is the Uiso parameter of the atom ``Ni0``. - -4. Create the refinement equation. +4. Run the refinement. .. code-block:: text - create equation variables s0 - set equation as "G1*s0" - -``G1`` is the identifier for the structure file loaded earlier. In the -equation, it represents the PDF data generated from the structure `G1`. -``s0`` is created to count for the scaling factor. - -5. Store the results. - -.. code-block:: text - - save to "output_results.json" - -The results of the refinement will be saved to a file -named ``output_results.json``. - -6. List variables to be refined. - -.. code-block:: text - - variables: - --- - - G1.a: 3.52 - - s0: 0.4 - - G1.Uiso_0: 0.005 - - G1.delta2: 2 - - qdamp: 0.04 - - qbroad: 0.02 - --- - -Only variables listed in this section will be refined during the -execution of the macro, and the variables will also be refined in that order. -Variables with initial values specified here will be used as the -starting point for the refinement. + solve( + profile_names=["ni_profile"], + model_names=["pdf"], + variable_names=[ + "pdf.phase.lattice.a", + "pdf.scale", + "pdf.phase.Ni0.Uiso", + "pdf.delta2", + "pdf.qdamp", + "pdf.qbroad", + ], + ) + +Only the variables listed in ``variable_names`` are refined. .. note:: - The naming of variables follows the format - ``structure_identifier.parameter``, - ``profile_parameter``, or - ``equation_parameter``. - - For parameters belonging to a specific atom in the parameter, - the naming follows the format ``structure_identifier.parameter_atomindex``. - e.g. ``G1.Uiso_0`` here refers to the Uiso parameter of the first atom in - the structure ``G1``. - - For constrained parameters, it will use the first parameter in the - constraint. e.g. Here, lattice parameter ``a=b=c``, - and ``Usio_0=Uiso_i, i=1,2,3``, ``a`` and ``Uiso_0`` are used as the - reference variables. + + All commands from in a macro file are also tools inside + ``cmi-agentic-ui`` MCP server as they are essentially wrappers around the + same underlying methods. Please refer to :doc:`diffpy-agent` + for more information. diff --git a/news/macro.rst b/news/macro.rst new file mode 100644 index 0000000..9b31bf8 --- /dev/null +++ b/news/macro.rst @@ -0,0 +1,23 @@ +**Added:** + +* Update the grammar of macro to be compatible with cmi-agentic-ui. + +**Changed:** + +* + +**Deprecated:** + +* + +**Removed:** + +* + +**Fixed:** + +* + +**Security:** + +* diff --git a/src/diffpy/apps/app_runmacro.py b/src/diffpy/apps/app_runmacro.py deleted file mode 100644 index 4ac47a7..0000000 --- a/src/diffpy/apps/app_runmacro.py +++ /dev/null @@ -1,298 +0,0 @@ -import inspect -from collections import OrderedDict -from pathlib import Path - -import yaml -from textx import metamodel_from_str - -from diffpy.apps.pdfadapter import PDFAdapter - -grammar = r""" -Program: - commands*=Command - variable=VariableBlock? -; - -Command: - LoadCommand | SetCommand | CreateCommand | SaveCommand -; - -LoadCommand: - 'load' component=ID name=ID 'from' source=STRING -; - -SetCommand: - 'set' name=ID attribute=ID 'as' value+=Value[eolterm] - | 'set' name=ID 'as' value+=Value[eolterm] -; - -CreateCommand: - 'create' 'equation' 'variables' value+=Value[eolterm] -; - -SaveCommand: - 'save' 'to' source=STRING -; - -VariableBlock: - 'variables:' '---' content=/[\s\S]*?(?=---)/ '---' -; - -Value: - STRICTFLOAT | INT | STRING | RawValue -; - -RawValue: - /[^\s]+/ -; - - -Comment: /#[^\n]*/; -""" - - -class MacroParser: - def __init__(self): - self.pdfadapter = PDFAdapter() - self.meta_model = metamodel_from_str(grammar) - self.meta_model.register_obj_processors( - { - "SetCommand": self.set_command_processor, - "LoadCommand": self.load_command_processor, - "VariableBlock": self.parameter_block_processor, - "CreateCommand": self.create_command_processor, - "SaveCommand": self.save_command_processor, - } - ) - # key: method_name.argument_name - # value: argument_value - self.inputs = {} - # key: structure name or profile name set in the macro - # value: 'structure' or 'profile' - self.variables = OrderedDict() - - def parse(self, code): - self.meta_model.model_from_str(code) - - def input_as_list(self, key, value): - if key in self.inputs: - if not isinstance(self.inputs[key], list): - self.inputs[key] = [self.inputs[key]] - else: - self.inputs[key].append(value) - else: - if isinstance(value, list): - self.inputs[key] = value - else: - self.inputs[key] = [value] - - def load_command_processor(self, command): - if command.component == "structure": - # TODO: support multiple structures input in the future - if ( - self.inputs.get("initialize_structures.structure_paths") - is not None - ): - raise ValueError( - "Multiple structures are not supported by `runmacro` yet. " - "Please use python script instead." - ) - key = "initialize_structures.structure_paths" - variable = "structure" - elif command.component == "profile": - if self.inputs.get("initialize_profile.profile_path") is not None: - raise ValueError( - "Multiple profiles are not supported by `runmacro`. " - "Please use python script instead." - ) - key = "initialize_profile.profile_path" - variable = "profile" - else: - raise ValueError( - f"Unknown component type: {command.component} " - "Please use 'structure' or 'profile'." - ) - source_path = Path(command.source) - if not source_path.exists(): - raise FileNotFoundError( - f"{command.component} {source_path} not found. " - "Please ensure the path is correct and the file exists." - ) - self.inputs[key] = str(source_path) - self.variables[command.name] = variable - if variable == "structure": - self.input_as_list("initialize_structures.names", command.name) - - def set_command_processor(self, command): - if command.name == "equation": - key = "initialize_contribution.equation" - elif command.name in self.variables: - if self.variables[command.name] == "structure": - if command.attribute == "spacegroup": - key = "initialize_structures.spacegroups" - else: - key = "initialize_structures." + command.attribute - elif self.variables[command.name] == "profile": - key = "initialize_profile." + command.attribute - else: - raise ValueError( - f"Unknown variable type for name: {command.name}. " - "This is an internal error. Please report this issue to " - "the developers." - ) - else: - raise ValueError( - f"Unknown name in set command: {command.name}. " - "Please ensure that it is typed correctly as 'equation' or " - "it matches a previously loaded structure or " - "profile. " - ) - self.input_as_list(key, command.value) - - def parameter_block_processor(self, variable_block): - self.inputs["set_initial_variable_values.variable_name_to_value"] = {} - self.inputs["refine_variables.variable_names"] = [] - parameters = yaml.safe_load(variable_block.content) - if not isinstance(parameters, list): - raise ValueError( - "Parameter block should contain a list of parameters. " - "Please use the following format:\n" - "- param1 # use default initial value\n" - "- param2: initial_value\n" - ) - for item in parameters: - if isinstance(item, str): - self.inputs["refine_variables.variable_names"].append( - item.replace(".", "_") - ) - elif isinstance(item, dict): - pname, pvalue = list(item.items())[0] - self.inputs[ - "set_initial_variable_values.variable_name_to_value" - ][pname.replace(".", "_")] = pvalue - self.inputs["refine_variables.variable_names"].append( - pname.replace(".", "_") - ) - else: - raise ValueError( - "Variables block items are not correctly formatted. " - "Please use the following format:\n" - "- param1 # use default initial value\n" - "- param2: initial_value\n" - ) - - def create_command_processor(self, command): - self.inputs["add_contribution_variables.variable_names"] = ( - command.value - ) - - def save_command_processor(self, command): - self.inputs["save_results.result_path"] = command.source - - def required_args(self, func): - sig = inspect.signature(func) - return [ - name - for name, p in sig.parameters.items() - if p.default is inspect.Parameter.empty - and p.kind - in ( - inspect.Parameter.POSITIONAL_ONLY, - inspect.Parameter.POSITIONAL_OR_KEYWORD, - inspect.Parameter.KEYWORD_ONLY, - ) - ] - - def call_pdfadapter_method(self, method_name, function_requirement): - func = getattr(self.pdfadapter, method_name) - required_arguments = self.required_args(func) - arguments = { - key.split(".")[1]: value - for key, value in self.inputs.items() - if key.startswith(method_name) - } - if not all(arg in arguments for arg in required_arguments): - missing_args = [ - arg for arg in required_arguments if arg not in arguments - ] - if function_requirement == "required": - raise ValueError( - "Missing required arguments for function " - f"'{method_name}' {', '.join(missing_args)}. " - "Please provide these arguments in the macro file." - ) - elif function_requirement == "optional": - print( - "Missing required arguments for function " - f"'{method_name}' {', '.join(missing_args)}. " - "This function will be skipped. " - "Please provide these arguments in the macro file " - "to activate this function." - ) - return - func(**arguments) - - def preprocess(self): - methods_to_call = [ - ("initialize_profile", "required"), - ("initialize_structures", "required"), - ("initialize_contribution", "required"), - ("initialize_recipe", "required"), - ("add_contribution_variables", "optional"), - ("set_initial_variable_values", "optional"), - ] - for method in methods_to_call: - self.call_pdfadapter_method(*method) - - def run(self): - methods_to_call = [ - ("refine_variables", "required"), - ("save_results", "optional"), - ] - for method in methods_to_call: - self.call_pdfadapter_method(*method) - return self.pdfadapter.get_results() - - -def runmacro(args): - dpin_path = Path(args.file) - if not dpin_path.exists(): - raise FileNotFoundError( - f"{str(dpin_path)} not found. Please check if this file " - "exists and provide the correct path to it." - ) - dsl_code = dpin_path.read_text() - parser = MacroParser() - parser.parse(dsl_code) - parser.preprocess() - return parser.run() - - -if __name__ == "__main__": - parser = MacroParser() - code = f""" -load structure G1 from "{str(Path(__file__).parents[3] / "tests/data/Ni.cif")}" -load profile exp_ni from "{str(Path(__file__).parents[3] / "tests/data/Ni.gr")}" - -set G1 spacegroup as auto -set exp_ni q_range as 0.1 25 -set exp_ni calculation_range as 1.5 50 0.01 -create equation variables s0 -set equation as "s0*G1" -save to "results.json" - -variables: ---- -- G1.a: 3.52 -- s0: 0.4 -- G1.Uiso_0: 0.005 -- G1.delta2: 2 -- qdamp: 0.04 -- qbroad: 0.02 ---- -""" # noqa: E501 - parser.parse(code) - parser.preprocess() - recipe = parser.pdfadapter.recipe - for pname, param in recipe._parameters.items(): - print(f"{pname}: {param.value}") diff --git a/src/diffpy/apps/apps.py b/src/diffpy/apps/apps.py index ead22c2..3ac2470 100644 --- a/src/diffpy/apps/apps.py +++ b/src/diffpy/apps/apps.py @@ -3,7 +3,7 @@ from rich.prompt import Confirm from diffpy.apps.app_agentify import agentify, list_skills -from diffpy.apps.app_runmacro import runmacro +from diffpy.apps.parametric_refinement.parametric_macro import runmacro from diffpy.apps.version import __version__ # noqa @@ -42,12 +42,12 @@ def main(): # runmacro application runmacro_parser = apps_parsers.add_parser( "runmacro", - help="Run a macro `<.dp-in>` file", + help="Run a macro file", ) runmacro_parser.add_argument( "file", type=str, - help="Path to the `<.dp-in>` macro file to be run", + help="Path to the macro file to be run", ) runmacro_parser.set_defaults(func=runmacro) # agentify application diff --git a/src/diffpy/apps/parametric_refinement/parametric_macro.py b/src/diffpy/apps/parametric_refinement/parametric_macro.py new file mode 100644 index 0000000..41703e9 --- /dev/null +++ b/src/diffpy/apps/parametric_refinement/parametric_macro.py @@ -0,0 +1,112 @@ +from pathlib import Path + +from textx import metamodel_from_str + +from diffpy.apps.parametric_refinement.refinement_session import ( + RefinementSession, +) + +grammar = r""" +Program: + calls*=Call +; + +Call: + name=ID '(' args*=Argument[','] ','? ')' +; + +Argument: + name=ID '=' value=Value | value=Value +; + +Value: + Bool | NoneValue | List | Dict | NUMBER | STRING +; + +Bool: + value=/True|False/ +; + +NoneValue: + value='None' +; + +List: + '[' items*=Value[','] ','? ']' +; + +Dict: + '{' items*=DictItem[','] ','? '}' +; + +DictItem: + key=STRING ':' value=Value +; + +Comment: + /#[^\n]*/ +; +""" + + +def to_python(node): + kind = type(node).__name__ + if kind == "Bool": + return node.value == "True" + if kind == "NoneValue": + return None + if kind == "List": + return [to_python(item) for item in node.items] + if kind == "Dict": + return {item.key: to_python(item.value) for item in node.items} + return node + + +class MacroRunner: + def __init__(self, session=None): + self.session = session if session is not None else RefinementSession() + self.meta_model = metamodel_from_str(grammar) + + def run(self, code): + calls = self.meta_model.model_from_str(code).calls + for call in calls: + if call.name.startswith("_") or not callable( + getattr(self.session, call.name, None) + ): + raise ValueError( + f"Unknown session method '{call.name}'. " + "Use a public method of RefinementSession." + ) + results = [] + for call in calls: + args = [] + kwargs = {} + for arg in call.args: + if arg.name: + kwargs[arg.name] = to_python(arg.value) + elif kwargs: + raise ValueError( + f"In '{call.name}', positional arguments must come " + "before keyword arguments." + ) + else: + args.append(to_python(arg.value)) + results.append(getattr(self.session, call.name)(*args, **kwargs)) + return results + + def run_file(self, path): + macro_path = Path(path) + if not macro_path.exists(): + raise FileNotFoundError( + f"{macro_path} not found. Please check the path to the " + "macro file." + ) + return self.run(macro_path.read_text()) + + +def runmacro(args): + results = MacroRunner().run_file(args.file) + for result in results: + if isinstance(result, dict) and "result" in result: + print(result["result"]) + return results diff --git a/src/diffpy/apps/pdfadapter.py b/src/diffpy/apps/pdfadapter.py deleted file mode 100644 index 1fe8276..0000000 --- a/src/diffpy/apps/pdfadapter.py +++ /dev/null @@ -1,342 +0,0 @@ -import json -import warnings -from pathlib import Path - -import numpy -from scipy.optimize import least_squares - -from diffpy.srfit.fitbase import ( - FitContribution, - FitRecipe, - FitResults, - Profile, -) -from diffpy.srfit.pdf import PDFGenerator, PDFParser -from diffpy.srfit.structure import constrain_as_space_group -from diffpy.structure.parsers import get_parser - - -class PDFAdapter: - """Build and configure a PDF fitting workflow backed by - diffpy.srfit. - - The adapter assembles the profile, structure generators, contribution, - and recipe objects required for a PDF refinement, then exposes the fit - results in a JSON-compatible dictionary. - - Public methods - -------------- - initialize_profile(profile_path: str, q_range=None, calculation_range=None) - Load the experimental PDF profile. - initialize_structures( - structure_paths: list[str], run_parallel=True, spacegroups=None, - names=None - ) - Load structures and create PDFGenerator objects for them. - initialize_contribution(equation_string=None) - Create the FitContribution from the loaded profile and generators. - initialize_recipe() - Create the FitRecipe for the current contribution. - set_initial_variable_values(variable_name_to_value: dict) - Set recipe parameter values by name. - get_results() - Return the current fit results as a JSON-compatible dictionary. - """ - - def initialize_profile( - self, - profile_path: str, - q_range=None, - calculation_range=None, - ): - """Load the experimental PDF profile. - - Parameters - ---------- - profile_path : str - The path to the experimental PDF profile file. - q_range : sequence of float, optional - The two-element sequence containing qmin and qmax. When omitted, - the values parsed from the profile file are used. - calculation_range : list or tuple of three floats, optional - The three-element sequence or mapping defining xmin, xmax, and dx - for the calculation range. When omitted, the range parsed from the - profile file is used. - """ - profile = Profile() - parser = PDFParser() - parser.parse_file(profile_path) - profile.load_parsed_data(parser) - if q_range is not None: - profile.meta["qmin"] = q_range[0] - profile.meta["qmax"] = q_range[1] - if calculation_range is not None: - if isinstance(calculation_range, (list, tuple)): - calculation_range = { - "xmin": calculation_range[0], - "xmax": calculation_range[1], - "dx": calculation_range[2], - } - profile.set_calculation_range(**calculation_range) - self.profile = profile - - def initialize_structures( - self, - structure_paths: list[str], - run_parallel=True, - spacegroups=None, - names=None, - ): - """Load structures and create PDFGenerator objects for them. - - This method should be called after initialize_profile. - - Parameters - ---------- - structure_paths : list of str - The paths to structure files in CIF format. - run_parallel : bool - If True, enable generator parallelization when the optional - dependencies are available. - spacegroups : list of str or None - The space groups are inferred from the parsed CIF - files when available and otherwise default to "P1". They will be - used to impose symmetry constraints on the structure parameters - during fitting. - names : list of str or None - The names assigned to the generators. Missing names default to - "G1", "G2", and so on. - """ - if isinstance(structure_paths, str): - structure_paths = [structure_paths] - structures = [] - have_spacegroups = False - if spacegroups is not None: - if len(spacegroups) != len(structure_paths): - raise ValueError( - f"spacegroups list {spacegroups} must match " - f"structure_paths list {structure_paths}. " - "Please provide a space group for each structure or set " - "spacegroups to None to infer them automatically." - ) - else: - have_spacegroups = True - if not have_spacegroups: - spacegroups = [] - pdfgenerators = [] - if run_parallel: - try: - import multiprocessing - from multiprocessing import Pool - - import psutil - - syst_cores = multiprocessing.cpu_count() - cpu_percent = psutil.cpu_percent() - avail_cores = numpy.floor( - (100 - cpu_percent) / (100.0 / syst_cores) - ) - ncpu = int(numpy.max([1, avail_cores])) - pool = Pool(processes=ncpu) - self.pool = pool - except ImportError: - warnings.warn( - "\nYou don't appear to have the necessary packages for " - "parallelization. Proceeding without parallelization." - ) - run_parallel = False - for i, structure_path in enumerate(structure_paths): - name = names[i] if names and i < len(names) else f"G{i+1}" - stru_parser = get_parser("cif") - structure = stru_parser.parse(Path(structure_path).read_text()) - sg = getattr(stru_parser, "spacegroup", None) - spacegroup = sg.short_name if sg is not None else "P1" - structures.append(structure) - if not have_spacegroups: - spacegroups.append(spacegroup) - else: - if spacegroups[i] == "auto": - spacegroups[i] = spacegroup - pdfgenerator = PDFGenerator(name) - pdfgenerator.setStructure(structure) - if run_parallel: - pdfgenerator.parallel(ncpu=ncpu, mapfunc=self.pool.map) - pdfgenerators.append(pdfgenerator) - self.spacegroups = spacegroups - self.pdfgenerators = pdfgenerators - - def initialize_contribution(self, equation=None): - """Create the FitContribution from the loaded profile and - generators. - - This method should be called after initialize_profile and - initialize_structures. - - Parameters - ---------- - equation : list of str, optional - The list of a single equation passed to - FitContribution.set_equation - - Returns - ------- - FitContribution - The configured contribution object. - """ - equation = equation[0] if equation is not None else None - contribution = FitContribution("pdfcontribution") - contribution.set_profile(self.profile) - for pdfgenerator in self.pdfgenerators: - contribution.add_profile_generator(pdfgenerator) - contribution.set_equation(equation) - self.contribution = contribution - return self.contribution - - def initialize_recipe( - self, - ): - """Create the FitRecipe for the current contribution. - - This method should be called after initialize_contribution. The - recipe includes shared qdamp and qbroad variables, per-generator - delta1 and delta2 variables, and structure parameters - constrained by the space group. - """ - recipe = FitRecipe() - recipe.add_contribution(self.contribution) - qdamp = recipe.create_new_variable("qdamp", fixed=False, value=0.04) - qbroad = recipe.create_new_variable("qbroad", fixed=False, value=0.02) - for i, (pdfgenerator, spacegroup) in enumerate( - zip(self.pdfgenerators, self.spacegroups) - ): - for pname in [ - "delta1", - "delta2", - ]: - par = getattr(pdfgenerator, pname) - recipe.add_variable( - par, name=f"{pdfgenerator.name}_{pname}", fixed=False - ) - recipe.add_constraint(pdfgenerator.qdamp, qdamp) - recipe.add_constraint(pdfgenerator.qbroad, qbroad) - stru_parset = pdfgenerator.phase - spacegroupparams = constrain_as_space_group( - stru_parset, spacegroup - ) - for par in spacegroupparams.xyzpars: - recipe.add_variable( - par, name=f"{pdfgenerator.name}_{par.name}", fixed=False - ) - for par in spacegroupparams.latpars: - recipe.add_variable( - par, name=f"{pdfgenerator.name}_{par.name}", fixed=False - ) - for par in spacegroupparams.adppars: - recipe.add_variable( - par, name=f"{pdfgenerator.name}_{par.name}", fixed=False - ) - recipe.fithooks[0].verbose = 0 - self.recipe = recipe - - def add_contribution_variables(self, variable_names): - """Add contribution parameters to the recipe. - - Parameters - ---------- - variable_names : list of str - The names of the variables to be added to the recipe. - e.g. 's0' for scale factor. - """ - for var_name in variable_names: - self.recipe.add_variable( - getattr(self.contribution, var_name), - name=var_name, - fixed=False, - ) - - def refine_variables(self, variable_names): - """Refine the specified variables. - - Parameters - ---------- - variable_names : list of str - The names of the variables to be refined. - """ - self.recipe.fix("all") - for var in variable_names: - self.recipe.free(var) - least_squares(self.recipe.residual, self.recipe.values) - - def set_initial_variable_values(self, variable_name_to_value: dict): - """Set recipe parameter values by name. - - Parameters - ---------- - variable_name_to_value : dict - Mapping from recipe variable names to new values. - """ - for vname, vvalue in variable_name_to_value.items(): - self.recipe._parameters[vname].set_value(vvalue) - - def get_results(self): - """Return the current fit results as a JSON-compatible - dictionary. - - Returns - ------- - dict - Residual statistics, variable values, constraints, the covariance - matrix, and a certainty flag. - """ - fit_results = FitResults(self.recipe) - results_dict = {} - results_dict["residual"] = fit_results.residual - results_dict["contributions"] = ( - fit_results.residual - fit_results.penalty - ) - results_dict["restraints"] = fit_results.penalty - results_dict["chi2"] = fit_results.chi2 - results_dict["reduced_chi2"] = fit_results.rchi2 - results_dict["rw"] = fit_results.rw - # variables - results_dict["variables"] = {} - for name, val, unc in zip( - fit_results.varnames, fit_results.varvals, fit_results.varunc - ): - results_dict["variables"][name] = { - "value": val, - "uncertainty": unc, - } - # fixed variables - results_dict["fixed_variables"] = {} - if fit_results.fixednames is not None: - for name, val in zip( - fit_results.fixednames, fit_results.fixedvals - ): - results_dict["fixed_variables"][name] = {"value": val} - # constraints - results_dict["constraints"] = {} - if fit_results.connames and fit_results.showcon: - for con in fit_results.conresults.values(): - for i, loc in enumerate(con.conlocs): - names = [obj.name for obj in loc] - name = ".".join(names) - val = con.convals[i] - unc = con.conuncs[i] - results_dict["constraints"][name] = { - "value": val, - "uncertainty": unc, - } - # covariance matrix - results_dict["covariance_matrix"] = fit_results.cov.tolist() - # certainty - certain = True - for con in fit_results.conresults.values(): - if (con.dy == 1).all(): - certain = False - results_dict["certain"] = certain - return results_dict - - def save_results(self, result_path): - with open(result_path, "w") as f: - json.dump(self.get_results(), f, indent=4) diff --git a/tests/test_pdfadapter.py b/tests/test_pdfadapter.py deleted file mode 100644 index 612435f..0000000 --- a/tests/test_pdfadapter.py +++ /dev/null @@ -1,54 +0,0 @@ -from pathlib import Path - -import numpy -from helper import run_ni_example -from scipy.optimize import least_squares - -from diffpy.apps.pdfadapter import PDFAdapter - - -def test_pdfadapter(): - # C1: Run the same fit with pdfadapter and diffpy_cmi - # Expect the refined parameters to be the same within 1e-5 - ni_refined_parameters = run_ni_example() - # diffpy_cmi fitting - structure_path = Path(__file__).parent / "data" / "Ni.cif" - profile_path = Path(__file__).parent / "data" / "Ni.gr" - initial_pv_dict = { - "s0": 0.4, - "qdamp": 0.04, - "qbroad": 0.02, - "G1_a": 3.52, - "G1_delta2": 2, - "G1_Uiso_0": 0.005, - } - variables_to_refine = [ - "G1_a", - "s0", - "G1_Uiso_0", - "G1_delta2", - "qdamp", - "qbroad", - ] - adapter = PDFAdapter() - adapter.initialize_profile( - str(profile_path), q_range=(0.1, 25), calculation_range=(1.5, 20, 0.01) - ) - adapter.initialize_structures([str(structure_path)]) - adapter.initialize_contribution(equation=["s0*G1"]) - adapter.initialize_recipe() - adapter.recipe.add_variable( - list(adapter.recipe._contributions.values())[0].s0 - ) - adapter.set_initial_variable_values(initial_pv_dict) - adapter.recipe.fix("all") - for var in variables_to_refine: - adapter.recipe.free(var) - least_squares(adapter.recipe.residual, adapter.recipe.values) - pdfadapter_pv_dict = {} - for pname, parameter in adapter.recipe._parameters.items(): - pdfadapter_pv_dict[pname] = parameter.value - for var_name in variables_to_refine: - pdfadapter_value = pdfadapter_pv_dict[var_name] - diffpy_value = ni_refined_parameters[var_name] - assert numpy.isclose(diffpy_value, pdfadapter_value, atol=1e-5) diff --git a/tests/test_runmacro.py b/tests/test_runmacro.py index a7e53ba..9b02279 100644 --- a/tests/test_runmacro.py +++ b/tests/test_runmacro.py @@ -1,201 +1,117 @@ -import re from pathlib import Path +from types import SimpleNamespace import numpy -import pytest from helper import run_ni_example -from diffpy.apps.app_runmacro import MacroParser - -_STRUCTURE_PATH = str(Path(__file__).parent / "data" / "Ni.cif") -_PROFILE_PATH = str(Path(__file__).parent / "data" / "Ni.gr") - - -def test_meta_model(): - # C1: Run the same fit with pdfadapter and diffpy_cmi - # Expect the refined parameters to be the same within 1e-5 - # diffpy_cmi fitting - variables_to_refine = [ - "G1_a", - "s0", - "G1_Uiso_0", - "G1_delta2", - "qdamp", - "qbroad", - ] - diffpy_pv_dict = run_ni_example() - - diffpy_dsl = f""" -load structure G1 from "{_STRUCTURE_PATH}" -load profile exp_ni from "{_PROFILE_PATH}" - -set G1 spacegroup as auto -set exp_ni q_range as 0.1 25 -set exp_ni calculation_range as 1.5 20 0.01 -create equation variables s0 -set equation as "s0*G1" +from diffpy.apps.parametric_refinement.parametric_macro import ( + MacroRunner, + runmacro, +) -variables: ---- -- G1_a: 3.52 -- s0: 0.4 -- G1_Uiso_0: 0.005 -- G1_delta2: 2 -- qdamp: 0.04 -- qbroad: 0.02 ---- -""" - diffpy_interpreter = MacroParser() - diffpy_interpreter.parse(diffpy_dsl) - diffpy_interpreter.preprocess() - interpreter_results = diffpy_interpreter.run() - for var_name in variables_to_refine: - diffpy_value = diffpy_pv_dict[var_name] - interpreter_value = interpreter_results["variables"][var_name]["value"] - assert numpy.isclose(diffpy_value, interpreter_value, atol=1e-5) +_DATA_DIR = Path(__file__).parent / "data" -@pytest.mark.parametrize( - "command_string, expected_inputs, expected_variables", - [ - # C1: load structure G1 from "path/to/structure.cif" - # Expect inputs and variables to be set correctly - ( - f'load structure G1 from "{_STRUCTURE_PATH}"', - { - "initialize_structures.structure_paths": _STRUCTURE_PATH, - "initialize_structures.names": ["G1"], - }, - {"G1": "structure"}, - ), - # C2: load profile exp_ni from "path/to/profile.gr" - # Expect inputs and variables to be set correctly - ( - f'load profile exp_ni from "{_PROFILE_PATH}"', - {"initialize_profile.profile_path": _PROFILE_PATH}, - {"exp_ni": "profile"}, - ), - # C3: create equation variables s0 - # Expect variable names to be stored - ( - "create equation variables s0", - {"add_contribution_variables.variable_names": ["s0"]}, - {}, - ), - # C4: save to "results.json" - # Expect result path to be stored - ( - 'save to "results.json"', - {"save_results.result_path": "results.json"}, - {}, - ), - # C5: set equation as "s0*G1" - # Expect equation to be stored - ( - 'set equation as "s0*G1"', - {"initialize_contribution.equation": ["s0*G1"]}, - {}, - ), - # C6: set exp_ni q_range as 0.1 25 - # Expect q_range to be stored - ( - f'load profile exp_ni from "{_PROFILE_PATH}"\n' - "set exp_ni q_range as 0.1 25", - {"initialize_profile.q_range": [0.1, 25]}, - {"exp_ni": "profile"}, - ), - # C7: set exp_ni calculation_range as 1.5 50 0.01 - # Expect calculation_range to be stored - ( - f'load profile exp_ni from "{_PROFILE_PATH}"\n' - "set exp_ni calculation_range as 1.5 50 0.01", - {"initialize_profile.calculation_range": [1.5, 50, 0.01]}, - {"exp_ni": "profile"}, - ), - # C8: set G1 spacegroup as auto - # Expect spacegroup to be stored - ( - f'load structure G1 from "{_STRUCTURE_PATH}"\n' - "set G1 spacegroup as auto", - {"initialize_structures.spacegroups": ["auto"]}, - {"G1": "structure"}, - ), - # C9: variables section with multiple variables - # Expect variable names and values to be stored - ( - """ -variables: ---- -- G1_a: 3.52 -- s0 -- G1_Uiso_0: 0.005 ---- -""", - { - "set_initial_variable_values.variable_name_to_value": { - "G1_a": 3.52, - "G1_Uiso_0": 0.005, - }, - "refine_variables.variable_names": ["G1_a", "s0", "G1_Uiso_0"], - }, - {}, - ), - # C10: comments. Expect no error is raised by textx - ( - f""" -# This is a comment -load structure G1 from "{_STRUCTURE_PATH}" -""", - { - "initialize_structures.structure_paths": _STRUCTURE_PATH, - "initialize_structures.names": ["G1"], - }, - {"G1": "structure"}, - ), - ], +def test_runmacro(tmp_path): + # C1: Refine a nested sine model from a macro file with the same + # calls as test_refine_sine in test_refinement_session.py + # Expect the refined parameters to be close to the true values + xarray = numpy.linspace(-numpy.pi, numpy.pi, 100) + yarray = numpy.sin(xarray) + macro_file = tmp_path / "sine.dp-in" + macro_file.write_text(f""" +# nested sine model +add_profile_from_arrays( + xarray={xarray.tolist()}, + yarray={yarray.tolist()}, + profile_name="sine_profile", ) -def test_command_processor( - command_string, expected_inputs, expected_variables -): - parser = MacroParser() - parser.parse(command_string) - for key, value in expected_inputs.items(): - assert parser.inputs[key] == value - assert dict(parser.variables) == expected_variables +add_equation_model(model_name="sub", equation_string="a*x") +add_equation_model(model_name="main", equation_string="A*sin(sub)") +combine_models(parent_model_name="main", child_model_names=["sub"]) +set_model_profile(model_name="main", profile_name="sine_profile") +set_variables_value(name_value_dict={{"main.A": 0.8, "main.sub.a": 0.5}}) +solve( + profile_names=["sine_profile"], + model_names=["main"], + variable_names=["main.A", "main.sub.a"], +) +""") + runner = MacroRunner() + runner.run_file(macro_file) + for name in ["main.A", "main.sub.a"]: + assert numpy.isclose( + runner.session.get_variable(name)["value"], 1.0, rtol=1e-2 + ) + # C2: Run the same macro file through the `runmacro` application entry + # Expect no error + runmacro(SimpleNamespace(file=str(macro_file))) + # C3: Run a macro that calls a private session method + # Expect a ValueError naming the method + try: + runner.run('_get_model("main")') + except ValueError as error: + assert "_get_model" in str(error) + else: + raise AssertionError("ValueError not raised") -@pytest.mark.parametrize( - "command_string, expected_exception, match", - [ - ( - f'load unknown foo from "{_STRUCTURE_PATH}"', - ValueError, - "Unknown component type: unknown " - "Please use 'structure' or 'profile'.", - ), - ( - 'load structure foo from "/nonexistent/path.cif"', - FileNotFoundError, - f"structure {Path('/nonexistent/path.cif')} not found. " - "Please ensure the path is correct and the file exists.", - ), - ( - f'load structure G1 from "{_STRUCTURE_PATH}"\n' - f'load structure G2 from "{_STRUCTURE_PATH}"', - ValueError, - "Multiple structures are not supported by `runmacro` yet. " - "Please use python script instead.", - ), - ( - f'load profile p1 from "{_PROFILE_PATH}"\n' - f'load profile p2 from "{_PROFILE_PATH}"', - ValueError, - "Multiple profiles are not supported by `runmacro`. " - "Please use python script instead.", - ), +def test_runmacro_ni(): + # C1: Refine Ni example using only a PDF model from a macro + # Expect the refined parameters are close to the ones + # obtained using diffpy.srfit script + runner = MacroRunner() + runner.run(f""" +add_profile_from_file( + profile_name="ni_profile", profile_path="{_DATA_DIR / "Ni.gr"}" +) +set_profile_calculation_range( + profile_name="ni_profile", xmin=1.5, xmax=20, dx=0.01 +) +update_profile_meta(profile_name="ni_profile", meta={{"qmin": 0.1}}) +add_pdf_model( + model_name="pdf", + structure_file_path="{_DATA_DIR / "Ni.cif"}", + finite=False, +) +constrain_pdf_model_space_group_symmetry( + model_name="pdf", space_group="Fm-3m" +) +set_variables_value( + name_value_dict={{ + "pdf.phase.lattice.a": 3.52, + "pdf.scale": 0.4, + "pdf.phase.Ni0.Uiso": 0.005, + "pdf.delta2": 2, + "pdf.qdamp": 0.04, + "pdf.qbroad": 0.02, + }} +) +solve( + profile_names=["ni_profile"], + model_names=["pdf"], + variable_names=[ + "pdf.phase.lattice.a", + "pdf.scale", + "pdf.phase.Ni0.Uiso", + "pdf.delta2", + "pdf.qdamp", + "pdf.qbroad", ], ) -def test_load_command_processor_bad(command_string, expected_exception, match): - parser = MacroParser() - with pytest.raises(expected_exception, match=re.escape(match)): - parser.parse(command_string) +""") + name_to_cmi_name = { + "pdf.scale": "s0", + "pdf.phase.lattice.a": "G1_a", + "pdf.phase.Ni0.Uiso": "G1_Uiso_0", + "pdf.delta2": "G1_delta2", + "pdf.qdamp": "qdamp", + "pdf.qbroad": "qbroad", + } + ni_refined_parameters = run_ni_example() + for name, cmi_name in name_to_cmi_name.items(): + assert numpy.isclose( + runner.session.get_variable(name)["value"], + ni_refined_parameters[cmi_name], + rtol=1e-2, + )