Need to parse large conf files

JaredBusch

I have a number of Asterisk configuration files that I need to parse in order to pull out some key highlights.

Someone tell me if I am off base on how to do this.....

All asterisk configurations files have headers in square brakcets [from-voipms] or [bobs-your-uncle], etc. These are called contexts.

After each context is lines of dial plan.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;   Bob's inbound 3145551212                      ;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
exten => 3145551212,1,NoOp()
same => n,Goto(main,123456,1)
same => n,Hangup()

What I need to do is parse the first line of each dial plan in each context
I would also like to get any comment up to 3 lines before the first line of each dial plan.
Comments always start with a ;
Dial plans always start with exten => + some numbers or text + ,1,

The underlying OS is Ubuntu 12.04.

Google tells me to make a file such as parseit.sh with this content

#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
    echo "Current line is: $line"
done < "$1"

Make it executable chmod +x parseit.sh
Then call it passing the file to parse as a parameter. ./parseit.sh extensions.conf

This is easy.

Now the need to create the logic to do what I said above.

I will certainly need to figure out some regex to determine the context and first line of dial plan. This part I am not strong at.

Getting the previous three lines is easy to put in the loop like this i think.

while
    $linem3 = $linem2;
    $linem2 = $linem1;
    $linem1 = $linem0;
    $linem0 = $line;
    if [ REGEX OF LINE MATCHES PATTERN ]
    then
        DO ALL THE THINGS (like echo to screen for testing).
    fi 
done

1337

Just for fun I found some extensions.conf file on the net to try. It led me to add some better formatting so the output is more readable.

PS. Just a small change like this that would have caused hours of search on regex, options for grep, awk, sed etc to try and make something work. That's why I think it's easier to just go full on programming language as soon as it is more complex than just finding a few lines that matches.

I know as this is all for naught but whatever.

Output from the php script:

File: sample.conf

[context_1]
; Bob's inbound 3145551212
exten => 3145551212,1,NoOp()

; Mary's inbound 4534535345
; With some added comments
exten => 4534535345,1,NoOp()

[context_2]
exten => 33333333,1,NoOp()

; What is this????????????
; Lets add a long comment section
; Line 3,,,
exten => 3145454,1,NoOp()

; Let's make a wonky comment line here
; And lets add this too
exten => 232342,1,NoOp()

[context_3]
exten => 7777777,1

exten => 8888888,1,NoOp()

PHP script v2

<?php
   // Ver 2 - improved formatting, only showing headings that have extensions
   
   // find matching files, put in array called filename
   $filename=glob("*.conf");
   

   // scan through all the files
   foreach ($filename as $f) {
      // print filename
      print "File: $f\n\n";

      // read all lines into lines array
      $lines=file($f);

      $no=0; // linenumber
      $comment=[]; // comments array
      $context=''; // context text

      // go through the file and pick out what we need
      foreach ($lines as $line) {
         $no++;

         // look for [context]
         if ($line[0]=='[') {
            $context=trim($line);
         }
         
         // look for extensions
         $search="exten =>";
         if (substr($line,0,strlen($search))==$search) {
            //print $line;

            // we found the extension row
            // lets take everything after "exten =>"
            $extline=trim(substr($line,strlen($search)));

            // lets divide it up
            $parts=explode(',', $extline);

            // check that it's ?????,1,?????
            if ($parts[1]=='1') {
               // valid extension, actual extension is $parts[0];
               $ext=trim($line);
               
               // print context if it has not been printed
               if ($context>'') { print "$context\n"; $context=''; }
               // print comments
               foreach($comment as $c) print "; $c\n";
               // print the extension and empty line
               print "$ext\n\n";
            }
            $comments=[]; // clear comments
         }
         
         // look for comments
         if ($line[0]==';') {
            // remove whitespace and ;
            $s=trim($line,"; \t\n\r\0\x0B");
            // add to comments if not empty but only first 3 lines
            if (($s>'') and (count($comment)<3)) $comment[]=$s;
         } else {

            // non empty line? => clear comments
            if (trim($line)>'') $comment=[];

         }
      }
   }


?>

Romo

Python has configparser which handles this files, INI files.

Maybe this would help you Jared - https://github.com/albfan/bash-ini-parser

1337

@JaredBusch

Just to clarify, what is it you want to read from the config files, is it just the extension number?
For example from exten => 3145551212,1,NoOp() to 3145551212 ?

What should the end result be? A new file with all the extension numbers?

JaredBusch

@Romo said in Need to parse large conf files:

Python has configparser which handles this files, INI files.

Maybe this would help you Jared - https://github.com/albfan/bash-ini-parser

Neither are very helpful.

JaredBusch

@Pete-S said in Need to parse large conf files:

@JaredBusch

Just to clarify, what is it you want to read from the config files, is it just the extension number?
For example from exten => 3145551212,1,NoOp() to 3145551212 ?

What should the end result be? A new file with all the extension numbers?

For now, I want the comments and the exten line.

Once i have it in a form I can analyze, i will be writing more statements to do specific things.

JaredBusch

Went and had some lunch. Getting back on this now.

Obsolesce

@JaredBusch said in Need to parse large conf files:

Went and had some lunch. Getting back on this now.

To me it seems easier to do in PowerShell. If I could get something together sometime, would you consider it? But I'll need more data to work with for testing.

JaredBusch

@Obsolesce said in Need to parse large conf files:

@JaredBusch said in Need to parse large conf files:

Went and had some lunch. Getting back on this now.

To me it seems easier to do in PowerShell. If I could get something together sometime, would you consider it? But I'll need more data to work with for testing.

What good would that do me when the OS is Ubuntu 12.04?

Obsolesce

@JaredBusch said in Need to parse large conf files:

@Obsolesce said in Need to parse large conf files:

@JaredBusch said in Need to parse large conf files:

Went and had some lunch. Getting back on this now.

To me it seems easier to do in PowerShell. If I could get something together sometime, would you consider it? But I'll need more data to work with for testing.

What good would that do me when the OS is Ubuntu 12.04?

My thought was that you could grab the file and run it on something with PS.

Or is this something that needs to run on the system? Perhaps it'll work with PS6.1?

scottalanmiller

@JaredBusch said in Need to parse large conf files:

@Obsolesce said in Need to parse large conf files:

@JaredBusch said in Need to parse large conf files:

Went and had some lunch. Getting back on this now.

To me it seems easier to do in PowerShell. If I could get something together sometime, would you consider it? But I'll need more data to work with for testing.

What good would that do me when the OS is Ubuntu 12.04?

Will PS install via AppImage in 12.04? That's so old, not sure what it can do.

1337

@JaredBusch

It's faster to do this programmatically than trying to piece it together with small utilities and scripting glue, especially if you want to store the results in a database or something like that.

I'll write some pseudo correct php code for you here to show what I mean.

// find matching files, put in array called filename
$filename:=glob("*.ini");

// scan through all the files
foreach ($filename as $f) {

   // read all lines into lines array
   $lines:=file($f);

   // go through the file and pick out what we need
   $line:=0; // linenumber
   foreach ($lines as $line) do {
      $line++;

      // check the start of each line to look for "exten =>"
      $search:="exten =>";
      if substr($line,0,strlen($search))==$search {
         // we found the extension row
         // lets take everything after "exten =>"
         $extline:=substr($line,strlen($search));
         // lets divide it up 
         $parts:=explode($extline, ",");
         $ext:=$parts[0]; // extension number

         // lets pick out the three preceeding lines
         $comment[1]:=$lines[$line-3];
         $comment[2]:=$lines[$line-2];
         $comment[3]:=$lines[$line-1];

         // do more stuff on the line we picked out here
        
         // lets save what we might need in an array
         $extensions[$ext]:=array("comments"=>$comment, "linenumber"=>$line, "file"=>$f);
      }
      
      // search for other stuff and what not
   }
   
   // do more stuff on the file
}

// remove comment to print array for inspection/debug
//print_r($extensions);

// lets print out all extension found in all files
foreach ($extensions as $ext => $data) do {
   print "Extension: $ext\r\n";
}

JaredBusch

@Pete-S said in Need to parse large conf files:

@JaredBusch

It's faster to do this programmatically than trying to piece it together with small utilities and scripting glue, especially if you want to store the results in a database or something like that.

I'll write some pseudo correct php code for you here to show what I mean.

// find matching files, put in array called filename
$filename:=glob("*.ini");

// scan through all the files
foreach ($filename as $f) {

   // read all lines into lines array
   $lines:=file($f);

   // go through the file and pick out what we need
   $line:=0; // linenumber
   foreach ($lines as $line) do {
      $line++;

      // check the start of each line to look for "exten =>"
      $search:="exten =>";
      if substr($line,0,strlen($search))==$search {
         // we found the extension row
         // lets take everything after "exten =>"
         $extline:=substr($line,strlen($search));
         // lets divide it up 
         $parts:=explode($extline, ",");
         $ext:=$parts[0]; // extension number

         // lets pick out the three preceeding lines
         $comment[1]:=$lines[$line-3];
         $comment[2]:=$lines[$line-2];
         $comment[3]:=$lines[$line-1];

         // do more stuff on the line we picked out here
        
         // lets save what we might need in an array
         $extensions[$ext]:=array("comments"=>$comment, "linenumber"=>$line, "file"=>$f);
      }
      
      // search for other stuff and what not
   }
   
   // do more stuff on the file
}

// remove comment to print array for inspection/debug
//print_r($extensions);

// lets print out all extension found in all files
foreach ($extensions as $ext => $data) do {
   print "Extension: $ext\r\n";
}

And does all of that work with PHP 5.3

JaredBusch

@Pete-S said in Need to parse large conf files:

@JaredBusch

It's faster to do this programmatically than trying to piece it together with small utilities and scripting glue, especially if you want to store the results in a database or something like that.

Since when is a bash script not doing something programmatically?

matteo nunziati

@JaredBusch sorry I'm missing something here... Isn't this something you can do with grep? If you know how your line begins you can ask grep to find that line and some context before and after that line.

Obsolesce

@matteo-nunziati said in Need to parse large conf files:

@JaredBusch sorry I'm missing something here... Isn't this something you can do with grep? If you know how your line begins you can ask grep to find that line and some context before and after that line.

Like each line that begins with ; and exten?

matteo nunziati

@Obsolesce if schema is:
[Blabla]
; Comment line1
; Comment line 2
; Comment line 3
exten =>

Yes you can ask for:
Grep -e "[.*]" -A 4

Or similar: I'm in bed with my smartphone atm... Dont' remember the synthax

Grrr squared brackets have to be escaped but I can't...

1337

@JaredBusch said in Need to parse large conf files:

@Pete-S said in Need to parse large conf files:

@JaredBusch

It's faster to do this programmatically than trying to piece it together with small utilities and scripting glue, especially if you want to store the results in a database or something like that.

Since when is a bash script not doing something programmatically?

Well, bash is programs of course a program but using grep for instance you are defining search patterns and using command line options to get it to pick out specific lines. It's not the same as writing a program that would go back and forth in the files and pick out what you need and put it together anyway you like.

1337

@JaredBusch said in Need to parse large conf files:

And does all of that work with PHP 5.3

In general yes (you just run it with php filename.php) but I'd have to have some files and debug it to know for sure. I just wrote it on the top of my head as an example just as I'm writing this.

You might favor another language. Python, java, javascript, what do I know.

JaredBusch

@matteo-nunziati said in Need to parse large conf files:

@JaredBusch sorry I'm missing something here... Isn't this something you can do with grep? If you know how your line begins you can ask grep to find that line and some context before and after that line.

The is what is needed.

There are many more contexts and many dial pans per context.

They are all (generally commented).

1337

@JaredBusch said in Need to parse large conf files:

@matteo-nunziati said in Need to parse large conf files:

@JaredBusch sorry I'm missing something here... Isn't this something you can do with grep? If you know how your line begins you can ask grep to find that line and some context before and after that line.

The is what is needed.

There are many more contexts and many dial pans per context.

They are all (generally commented).

So basically all extension numbers and all preceding comment lines for those extensions under each [context]?

And what should the end result look like?

Solved Need to parse large conf files