This action might not be possible to undo. Are you sure you want to continue?
Our client will open an account with us so he can store all his invoices. We have 02 types of invoices: the one issued by our client to another company and the one received by our client from another company. We will store both types in our repository. All invoices are sent in XML format (*) and per e-mail.
(*) Please, check the RAR I sent you. It contains several XML’s so you can analyze the structure. You will see that all of them have the same scheme.
This e-mail can be sent by any company issuing invoices to our client OR it can be sent by our client himself when issuing an invoice to another company. In these e-mails we can have the XML’s attached but also ZIP’s or RAR’s with several XML’s inside. In the future we will develop a backend interface where the client will have the option of uploading files manually if desired. In this document I will explain the steps we need to follow in order to retrieve the XML’s from the e-mails, make some verifications, store part of its data in database-tables and store its entire content in a database field so we can delete the file but recover its content if needed in the future. 1.2. Scalability It’s also very important that we plan a code and database structure so that we can scale the capacity of handling the files (recover from e-mail, unzip, read, verify, store data and so on) and storing its content. We intent to develop a system and offer it as a service (SaaS). So it’s going to be offered on-line for a large number of companies. It’s also important that you suggest ways of controlling and monitoring the processes so we can know if any problem arises when handling requests. The steps described in this document can suffer modifications in the future by changing some of its details but also by adding new steps in between. So we have to be prepared for that as well.
1.3. Technical Details We will develop the system using PHP. The database will be MySQL (free version). In the future we might need to migrate to a paid version of MySQL that can handle more data and more requests. We also thought about coding with ODBC so if we decide to migrate it to MS SQL we wouldn’t have problems. But I made some researches and saw that MySQL (paid versions) is already being used by huge sites like Zappos, Ticketmaster, Booking.com and many others. So, perhaps we don’t need to worry about using another DB. I would like to hear your opinion on that matter. 2. Process Details 2.1. Step 01 – Getting Attachment from E-mails Each client using our service will have an e-mail address so they can forward their XML’s to be handled and properly stored. These addresses will be as follows: ClientCNPJ@domain.com “ClientCNPJ” is the unique federal number for a company in Brazil. Example: 07.742.225/0001-62 and the e-mail would be in this case email@example.com It always uses this format: xx.xxx.xxx/xxxx-xx We came up with this idea but if you see a better solution, please, fell free to suggest. Our main concerns are: - Not to be spammed. Which would happen if we had only one e-mail address to all our clients - To be able to easily block a client from using our service if he decides not to pay the monthly fee. We would delete the email address from the server. So, the first step will be to get the e-mails and check for attachments. If an e-mail doesn’t have an attachment it can be deleted straightaway. If it does have one, we need to check the format. We should have a table of allowed formats in case we need to add new formats of compressed files like ZIP and RAR. For now we’ll have only XML, ZIP and RAR. All the rest can be deleted. >> So, if the e-mail contains one or more files with the allowed format we then go to step 2. (**) We also have thought about having an algorithm or something
that allow us to define priorities. I’m not sure if this should be defined on the first step or somewhere else. But the idea is to give priority to clients handling a higher volume of XML’s. We will offer the services through plans according to the volume of XML’s the company produces per month. I would like your opinion/suggestion on this issue as well. 2.2. Step 2 – Handling Files and their integrity We thought about saving all XML’s gotten from the e-mails in a “processing folder” so the script can, one by one, process the files. Please, see if it’s needed to have a previous folder where we would save the compressed files so they can be uncompressed before being transfer to this “processing folder”. Once the XML’s are extracted from possible ZIP’s and RAR’s we will have to check if the files are valid. For that we’ll check a few information that if not available in the files will require us to send an e-mail back to the sender to inform him. A copy of this alert will have to be sent to our client if he’s not the issuer of the invoice. I can after give you more information about this validation but basically we need to check if the main fields (or tags) are available in the file. I will send the list of data that needs to be checked. If the file contains them all, it’s valid. If one is missing, it’s not valid. Also, since every XML is an invoice, each one is generated by a company to another company. So we have the “issuer” and the “receiver”. Our client can “generate” invoices but can also have invoices generated to him. In the XML we have a TAG that informs 1 if our client is the “issuer” and 0 if our client is the “receiver”, so: a) If 0 the “companycpnj” used on the e-mail address has to be equal to the “companycnpj” of the “receiver”. Otherwise it would mean the person that sent the e-mail has sent it to the wrong e-mail address. b) If 1 the “companycpnj” used on the e-mail address has to be equal to the “companycnpj” of the “issuer”. Otherwise it would mean that our client has sent the XML to the wrong email address which is unlikely to happen. The verification above also needs to be done. 2.3. Step 3 – Extracting data and DB insert Once the file is valid we will start extracting data from it and input them in to the database tables. I will also send the tables we’ll need
for the project. Every XML is an invoice. So it’s generated by a company to another company. So we have the “issuer” and the “receiver”. We’ll have a table in the database for “companies” in general. So, we need to extract the data from both companies (which I will detail afterwards) and register them in the system’s database. But first we’ll have to check if these companies are not already registered in our DB. If yes, ok, if not, we need to make the registration. We also will register this XML in the XML’s Table in the DB. Please, see DB structure further on. 2.4. Step 4 – On-line Validation via Webservice There is a federal service that allows us to verify if a XML is officially valid. We need to connect to the webservice (SOAP), send a few parameters from the XML and the webservice will return if it’s a valid XML or not. This process will have to be repeated once again after 30 days which is the timeframe that the “issuer” has to cancel the invoice. So, we need to make sure the “issuer” has not canceled the invoice. But this revalidation will have to be a separated step. 2.5. Step 5 – Issuing a request protocol We thought about issuing a protocol each time a request (sent per email) is processed and successfully concluded. It will have to be a unique number generated by the system. We would have to send an e-mail with the protocol to the e-mail address that made the request (CC: our client) and with the list of XML’s handled and the referring “Status” (*) (*) We thought about having a few status so we can make it easier to inform the issuer and/or our client about the status of the XML’s processed. Examples: 1 – Concluded 2 – Integrity issues 3 – Invalid according to the Federal Webservice 4 – Concluded but to be revalidated after 30 days 5 – Invalid “issuer” 6 – Invalid “receiver”
Of course we need to find better terms for our status but that’s the idea. So, with the protocol we would send the list of XML’s handled in the request with the referring status of each XML. One protocol has to be issued for each e-mail received and processed. It doesn’t matter if the e-mail had one XML or a ZIP with 300 XML’s. The protocols and all its data will have to be kept in a table (DB) so we can recover later on. Who gets the protocol? If it’s a XML type 1 (*) only our client (account owner) receives it. If it’s a XML type 0 (*) our client and the “issuer” receives it. (*) As explained in step 2, in type 1 our client is the “issuer” of the invoice (XML) and in type 0 our client is the “receiver” of the invoice (XML). 3. Database Structure 3.1. Table 1 – Companies This is where we’ll keep the companies registered. They can be “issuer” or “receiver” of XML’s. >> Issuer:
- <emit> <CNPJ>66573759000171</CNPJ> (C 14) <xNome>ELETROLESTE COM IMP MAT ELETR LTDA</xNome> (C 60) <xFant>ELETROLESTE</xFant> (C 60) - <enderEmit> <xLgr>AV: ARICANDUVA</xLgr> (C 60) <nro>5200</nro> (C 60) <xBairro>JD ARICANDUVA</xBairro> (C 60) <cMun>3550308</cMun> (C 7) <xMun>SAO PAULO</xMun> ( C 60) <UF>SP</UF> (C 2) <CEP>03490000</CEP> ( C 8) <cPais>1058</cPais> (C 4) <xPais>BRASIL</xPais> (C 60) <fone>1127235000</fone> (C 10) </enderEmit> <IE>113190045117</IE> (C 14) <IM>ISENTO</IM> (C 15) <CNAE>4673700</CNAE> (C 7) <CRT>3</CRT> ( C 1) </emit>
- <dest> <CNPJ>51258044000157</CNPJ> (C 14)
<xNome>NASE COML ELETR E HIDR LTDA - 6173</xNome> (C 60) - <enderDest> <xLgr>AV TABELIAO PASSARELLA, 250</xLgr> (C 60) <nro>.</nro> (C 60) <xBairro>CENTRO</xBairro> (C 60) <cMun>3528502</cMun> (C 7) <xMun>MAIRIPORA</xMun> (C 60) <UF>SP</UF> (C 2) <CEP>07600000</CEP> (C 8) <cPais>1058</cPais> (C 4) <xPais>BRASIL</xPais> (C 60) <fone>1146044445</fone> (C 10) </enderDest> <IE>433019804111</IE> (C 14) <email>firstname.lastname@example.org</email> (C 60) </dest>
3.2. Table 2 – Invoices >> Client: First field is the client ID which will be “companycnpj” so that we can easily make a query to list all XML’s from a specific client. Also because we need to know who is the XML owner. This because we will charge our service based on the amount of XML stored for a client. It can be that for one XML both “issuer” and “receiver” are our client. So, we will need to know who needs to pay for that XML. But I’m not sure what to do if both clients send us the same XML one being the “issuer” and the other being the “receiver”. We would have duplicated content. Any suggestion? >> Basic XML Info:
- <ide> <cUF>35</cUF> (C 2) <cNF>00061268</cNF> (C 9) <natOp>Vendas de Mercadorias</natOp> (C 60) <mod>55</mod> (C 2) <serie>1</serie> (C 3) <nNF>61268</nNF> (C 9) <dEmi>2011-05-25</dEmi> (D 10 AAAA-MM-DD) <dSaiEnt>2011-05-25</dSaiEnt> (D 10 AAAA-MM-DD) <tpNF>1</tpNF> (C 1) <cMunFG>3550308</cMunFG> (C 7) <emit> <CNPJ>66573759000171</CNPJ> <dest> <CNPJ>51258044000157</CNPJ> infNFe versao="2.00" (C 4) Id="NFe35110566573759000171550010000612681000612681" (C 50) xmlns="http://www.portalfiscal.inf.br/nfe"> (C 60)
>> All XML Data: Then we would have ONE field in this table so we can keep the entire XML content. It has to be a LONGBLOB or a LONGTEXT field type. An XML can have up to 300kb of data. 4. Additional Comments This is basically what we need for now. We intent to develop a backend so our client can access reports, recover XML’s and stuff. But for now we need this “engine” to work. We are looking for a partner to develop the hard code of our system. This core will grow and will be adapted over time. So we are looking for a long-term relationship. Also, everything that was described represents the idea that we have in mind. We don’t have the technical skills to say that this is the best way to do it considering scalability, security and so on. So, we need you to analyze the entire idea and help us built this engine so we can achieve our goals. We intent to have thousands of clients handling at least 10.000 XML’s a year each. And we also would like to be able to have a flexible core so we can further develop the system and its engine. The code you guys produce needs to be very well documented too so our programmers can make small adjustments and maintenance if needed. We would appreciate if you can send us an offer detailing how do you intent to develop this engine, how each step described will be handle considering scalability issues and flexibilities allowing further development.
This action might not be possible to undo. Are you sure you want to continue?